# NSA Prefill CP Page-Aligned Cache Contract

This note records the contract we need before continuing MLA/index partial
reuse under CP shared KV + HiCache + EAGLE/MTP.

The immediate trigger was a production hang where the prefill process stopped
after target-layer MLA partial current reuse:

```text
/mnt/beegfs/cjy/sglang_cp_hicache_20260528_161408.log

layer=2
prefix_lens=[13184]
extend_lens=[37024]
page_size=64
prefix_pages=206
suffix_slots=579
current_rows=37024
kv_rows=87328
used_prefetch=True
```

`suffix_slots * page_size = 579 * 64 = 37056`, but only `37024` current
rows were appended.  The fast path consumed prefetched prefix pages and appended
only real current rows, so the last suffix page had `32` rows of page slack with
no explicit valid/invalid contract.

The target design is **not** to fall back for these short or unaligned suffixes.
Instead, all physical cache paths should share one page-aligned contract while
logical token semantics stay unpadded.

## Contract

Use two lengths everywhere the path crosses a logical/physical boundary:

```text
valid_tokens     = real request/cache tokens
padded_pages     = ceil(valid_tokens / page_size)
padded_tokens    = padded_pages * page_size
padding_tokens   = padded_tokens - valid_tokens
```

### Logical layer: always valid-token based

The following must continue to use `valid_tokens`:

- request token length,
- `seq_lens_cpu`,
- `extend_seq_lens_cpu`,
- radix tree key/logical node length,
- cache-hit logical length exposed to scheduler,
- decode position and max-context checks,
- EAGLE/MTP verification token count,
- attention-visible sequence length.

Padding tokens must not become request tokens.

### Physical layer: always page-aligned

The following should use `padded_pages` / `padded_tokens`:

- GPU KV pool allocation and owner-lane capacity,
- draft KV mirror allocation,
- HiCache host reservation,
- backup/load/evict accounting,
- disaggregated target/draft transfer page lists,
- CP shared-KV MLA/index dense materialization buffers,
- CP owner-lane capacity planning and snapshots.

Padding rows may exist in temporary dense buffers or physical pages, but they
must be invisible to attention and radix semantics.

### Visibility rule

Every remap path must preserve this invariant:

```text
logical loc maps to a dense row only if it is a real valid token.
padding/slack locs map to -1.
```

Padding KV rows can be zero-filled.  Correctness comes from masking/remapping
them out, not from their value.

### Kernel ownership

Production hot paths should use `tai-kernel` for page-padded compose/remap and
copy-style operations.  Python/Torch implementations are acceptable as CPU unit
test references and explicit warning fallbacks only; they should not be the
normal CUDA runtime path.

For MLA partial-current reuse, the intended runtime primitive is:

```text
preallocated slot-dense KV pages
+ current suffix KV copied into its slot-dense rows
+ zero padding rows left invisible
-> same dense KV buffer + mixed loc remap + current mask
```

This should be one `tai_kernel.nsa_prefill.cp_shared_kv_materialize` primitive,
not a sequence of `torch.sort/searchsorted/cat/zeros` operations in the
attention hot path.  If the TAI primitive is unavailable, SGLang must emit a
visible warning fallback rather than silently taking the slow path.

The TAI primitive should extend the existing materialize/remap kernel family:

- copy each valid `current_locs[i]` row into `page_inverse[page] * page_size + offset`;
- build a current-page inverse from page-grouped current locs;
- keep real current locs visible by exact loc check;
- map non-current rows inside current pages to `-1`.

This avoids the previous compact-current append path where tail-page slack had
slot-dense locs but no valid current KV row.

## Root-cause record: unsafe partial-current prefetch

The original MLA partial-current prefetch path had a narrower contract than the
page-level metadata it consumed.

Evidence:

- `cp_shared_kv_prefetch.py::consume_prefix_with_current()` says the prefetched
  prefix is already materialized while current/suffix pages are not copied from
  the shared pool; it then appends `current_kv_cache` directly.
- `cp_shared_kv_runtime.py::merge_materialized_and_current_kv()` only remaps
  entries present in `current_locs`; all non-current entries keep the existing
  `materialized_dense_locs`.
- `nsa_backend.py` called `consume_prefix_with_current()` whenever an MLA
  prefetcher existed on a mixed prefix/current batch, before falling back to the
  older partial materialize path.

That is safe only when the page-level suffix represented by the page table is
exactly covered by `current_locs`.  It is not safe when:

```text
ceil(extend_len / page_size) * page_size > extend_len
```

The older non-prefetch partial path looked safer because it materialized all
non-current logical locations and only substituted rows that matched
`current_locs`, but it still re-entered the compact
`merge_materialized_and_current_kv()` contract.  Under the page-aligned cache
contract this is no longer an acceptable fallback for mixed prefix/current MLA
reuse.

The fix is intentionally not a permanent fallback.  The page-slot prefetch
compose path creates a page-padded current suffix buffer and maps suffix slack
to `-1`.  If that compose path is unavailable, the runtime now fails fast with
`[CP_SHARED_KV_FAIL_FAST][mla_partial_current_prefetch]` instead of silently
using compact materialize/current merge.

## Existing evidence in the codebase

### Page-aligned split already treats tail as a page unit

`build_page_aligned_in_seq_split_list()` computes:

```python
full_pages = extend_len // page_size
tail_tokens = extend_len % page_size
num_page_units = full_pages + (1 if tail_tokens > 0 else 0)
```

The Phase 4 doc explicitly states that the tail unit is not split across CP
segments.  However, the current `split_list` still sums to `extend_len`, not
`padded_tokens`.  That is correct for compute tokens, but it means later cache
paths must not assume `split_list` size equals physical page coverage.

### HiCache metadata currently requires page-aligned logical_len

`CpHiCacheNodeMetadata.__post_init__()` rejects `logical_len % page_size != 0`,
and `reserve_write_cp()` rejects non-page-aligned `device_indices`.

This is already a physical-cache assumption.  If logical radix length remains
valid-token based, the metadata needs a clearer name/split:

```text
valid_len          # radix-visible logical length
padded_len         # physical cache/host metadata length
page_owners        # length padded_len / page_size
owned_positions    # local positions inside padded_len
```

Without this split, node split/load checks can accidentally force logical cache
semantics to be page-aligned.

### Owner-lane accounting is already page-derived

`HiRadixCache._cp_owner_token_counts()` counts `page_size` tokens per owner page.
The no-collective capacity plan already describes a per-owner page ledger.  This
means the capacity side is structurally ready for page-based accounting, but all
inputs to the ledger must be padded-page based.  A valid-token scalar is not
enough for CP owner-lane admission.

`cp_shared_kv_compute_owner.build_in_seq_page_compute_owners()` should therefore
return owners for every physical page unit, including short chunks where some
zigzag segments own zero pages.  Falling back only because `num_pages < 2 *
cp_size` breaks the page-owner lane contract and can force later materialize or
HiCache paths back to token-balanced behavior.

This owner assignment remains page-count based; `out_cache_loc` remains
valid-token based.  `alloc_extend_compute_owner()` uses the owner list to choose
whole physical pages, then `alloc_extend_naive()` returns exactly the valid
extend token locs.  The padded tail rows exist because the selected logical page
exists, not because fake token locs are appended to `out_cache_loc`.

### Load must replay page owners

`HiCacheController.load_cp()` reconstructs device allocation via
`alloc_pages_with_owners(page_owners)`.  This confirms that load-back is already
page-pattern based and should stay physical-page based.

## Impact surface

### 1. CP split and metadata

Files:

- `python/sglang/srt/layers/attention/nsa/utils.py`

Needed changes:

- keep `split_list` and `actual_token_count` valid-token based;
- add explicit padded page metadata to `NSAContextParallelMetadata` or an
  attached helper object;
- keep `segment_page_starts/ends` page-based, including tail page coverage;
- add a helper that returns `valid_tokens`, `padded_tokens`, `padded_pages`, and
  `padding_tokens` for the current extend/cache-hit suffix.

Risk:

- changing `split_list` to padded tokens would leak fake tokens into compute and
  logits paths.  Do not do that.

### 2. MLA partial current reuse

Files:

- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py`
- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_prefetch.py`
- `python/sglang/srt/layers/attention/nsa_backend.py`

Needed changes:

- use the existing page-slot dense buffer for mixed prefix/current reuse;
- copy valid current suffix rows into their slot-dense page rows with
  `tai-kernel` in the CUDA path;
- keep padding rows in the tail suffix page physically present but zero/unused;
- remap valid current locs to their slot-dense rows;
- map suffix slack/future locs to `-1`;
- keep existing full/current-only path behavior unchanged.

Short-term safe gate if this cannot be implemented atomically:

- WARNING fallback to the older partial materialize path when suffix slack is
  detected.  This is a stopgap only; the target contract is page-padded compose,
  not fallback.

### 3. NSA index partial reuse

Files:

- `python/sglang/srt/layers/attention/nsa/nsa_indexer.py`
- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_prefetch.py`

Needed changes:

- apply the same valid/padded contract as MLA;
- index page metadata can be reused across layers, but returned dense page/table
  remaps must not expose padding locs as valid;
- current index reuse should not assume that all locations in a suffix page are
  current tokens.

Risk:

- index is page-first by nature, so it is easy to accidentally treat the whole
  tail page as visible.  Tests must include non-page-aligned suffixes.

### 4. HiCache host metadata and radix split

Files:

- `python/sglang/srt/mem_cache/hiradix_cache.py`
- `python/sglang/srt/managers/cache_controller.py`

Needed changes:

- distinguish radix-visible `valid_len` from physical `padded_len`;
- store `page_owners` for `padded_len / page_size` pages;
- keep split rules page-aware: a radix split inside an in-flight or host-backed
  padded page must either split at a page boundary or keep the child pending
  until the physical operation is committed;
- make `_node_host_len()` / host-hit accounting return valid length to the
  scheduler but use padded length for load/evict/host free.

Risk:

- if `host_hit_length` becomes padded, scheduler may believe fake tokens were
  cached.  If physical metadata uses valid length, host reservation/load can
  under-allocate the tail page.

### 5. Capacity and owner-lane admission

Files:

- `python/sglang/srt/mem_cache/hiradix_cache.py`
- `python/sglang/srt/mem_cache/allocator.py`
- `python/sglang/srt/managers/cache_controller.py`

Needed changes:

- all required capacity should be expressed as pages or per-owner padded tokens;
- owner-lane vectors should be derived from padded `page_owners`, not valid
  token counts;
- target and draft reservation must use the same padded page pattern;
- no hot-path collective should be introduced to discover the mismatch.

Risk:

- if target uses padded capacity and draft uses valid capacity, draft can miss
  the tail page while target succeeds, causing cache-hit accept-rate corruption.

### 6. Disaggregated transfer

Files:

- `python/sglang/srt/disaggregation/prefill.py`
- `python/sglang/srt/disaggregation/decode.py`
- `python/sglang/srt/disaggregation/mooncake/conn.py`
- `python/sglang/srt/disaggregation/nixl/conn.py`

Needed changes:

- transfer page lists remain page-based and may include the padded tail page;
- request metadata sent across disaggregation must keep valid token length;
- decode should not infer real prompt length from transferred padded page count;
- draft payload follows target pages, not independent draft token length.

Risk:

- page transfer count and logical request length diverge by up to
  `page_size - 1` tokens.  Any path that uses one as the other is wrong.

### 7. EAGLE/MTP draft KV

Files:

- `python/sglang/srt/speculative/eagle_worker.py`
- `python/sglang/srt/speculative/eagle_info.py`
- `python/sglang/srt/managers/cache_controller.py`

Needed changes:

- draft physical pages mirror target padded pages;
- draft valid length mirrors target valid length;
- draft cannot independently choose a shorter valid-only capacity path;
- draft cache-hit prefetch/current reuse must use the same visibility mask.

Risk:

- draft and target can both be physically page-aligned while disagreeing on
  which rows are valid.  That still breaks verification/accept rate.

## Missing work / correction ledger

This section is the working checklist for the current implementation.  Each item
records what is still missing or what existing code needs to be corrected before
we can treat the page-aligned contract as complete.

### C1. Add an explicit page extent object

Current state:

- `build_page_aligned_in_seq_split_list()` computes `num_page_units` locally.
- Callers still pass raw lengths such as `extend_len`, `logical_len`, and
  `required_slots` without naming whether they are valid-token or padded-page
  quantities.

Correction:

- Add a small helper/data object, for example:

  ```text
  PageAlignedCacheExtent(
      valid_tokens,
      padded_pages,
      padded_tokens,
      padding_tokens,
  )
  ```

- Use it where the code crosses logical/physical boundaries.
- `valid_tokens=100, page_size=64` must produce `padded_pages=2` and
  `padded_tokens=128`; it must not depend on `cp_size`.

Tests:

- Unit-test exact examples such as `0`, `64`, `100`, and `128` valid tokens.
- Add an assertion that `100` tokens with `page_size=64, cp_size=8` does not
  become `1024` physical tokens.

### C2. Extend CP split metadata without padding `split_list`

Current state:

- `split_list` is correct because it sums to valid tokens.
- `PageAlignedInSeqSplitInfo` and `NSAContextParallelMetadata` expose only
  `page_aligned`, `page_size`, `extend_prefix_len`, and segment page ranges.
- There is no explicit field carrying padded suffix coverage.

Correction:

- Keep `split_list`, `actual_token_count`, and query compute valid-token based.
- Add explicit metadata such as `extend_valid_tokens`, `extend_padded_pages`,
  `extend_padded_tokens`, and `extend_padding_tokens`.
- Keep `segment_page_starts/ends` page-based and include the tail page coverage.
- Do not change `cp_split_and_rebuild_data()` to consume fake padding tokens.

Tests:

- `extend_len=100, page_size=64, cp_size=8` should have:
  - `sum(split_list) == 100`;
  - `extend_padded_pages == 2`;
  - `extend_padded_tokens == 128`;
  - zero-token segments for ranks/zigzag segments without pages.

### C3. Remove short-suffix fallback completely, but only as metadata padding

Current state:

- The old `num_pages < cp_size` / `num_pages < 2 * cp_size` fallback was removed
  from the split and compute-owner helpers.
- `should_use_replicated_compute_for_short_radix_hit()` now returns `False`.
- `can_cp_split()` still has generic CP minimum-token checks; those are compute
  feasibility checks, not page-padding rules.

Correction:

- Keep the principle: no fallback solely because a suffix has fewer than
  `cp_size` or `2 * cp_size` pages.
- Padding means "cover the partial tail page physically"; it does not mean
  "round up to cp_size pages".
- If a truly tiny batch cannot run through CP compute, the reason must be a
  clearly named compute limitation, not `too_short_for_page_aligned`.

Tests:

- `extend_len=100, page_size=64, cp_size=8` should use two physical page units.
- `extend_len=3 pages, cp_size=8` should produce three compute owners, not
  eight or sixteen.

### C4. MLA partial-current prefetch compose is fail-fast without prefetch

Current state:

- `CpSharedKVMlaPrefetcher.consume_prefix_with_current()` now uses a page-slot
  compose helper instead of compact-appending current KV rows.
- `fill_current_kv_page_slots_and_remap_locs()` copies valid current rows into
  their dense page slots and masks non-current locs in current pages to `-1`.
- A TAI primitive exists for the CUDA hot path, with a visible warning fallback
  to the Torch reference when unavailable.
- `nsa_backend.py` no longer uses the compact
  `merge_materialized_and_current_kv()` helper when MLA partial-current reuse
  has prefix + current suffix but page-slot prefetch compose is unavailable.
  That path now raises
  `[CP_SHARED_KV_FAIL_FAST][mla_partial_current_prefetch]` with layer/rank,
  prefix/extend lengths, table shape, and remap capacity metadata.

Design decision:

- Mixed prefix/current MLA reuse must use the page-slot compose path.  If the
  prefetcher is missing or `consume_prefix_with_current()` cannot provide a
  composed buffer, failing immediately is safer than re-entering the compact
  materialize/current merge path.
- Keep `merge_materialized_and_current_kv()` documented as compact-materialize
  only; do not silently use it for page-slot prefetch compose.

Tests:

- Existing MLA slack test covers a tail page where only part of the suffix page
  is valid.
- `test_mla_partial_current_path_fails_fast_instead_of_compact_fallback`
  statically guards `nsa_backend.py` against reintroducing
  `merge_materialized_and_current_kv()` calls and requires the fail-fast marker.
- CUDA execution of the TAI primitive must be verified remotely; do not test
  CUDA locally.

### C5. NSA index partial reuse still needs an equivalent padding contract

Current state:

- `CpSharedKVIndexPrefetcher.consume()` materializes suffix page slots into a
  dense page buffer and returns dense pages/block tables.
- Index prefetch is page-first, so it can easily treat a whole tail page as
  visible unless query/cache sequence lengths keep the tail rows masked.
- There is no dedicated unit test proving tail-page slack is invisible for NSA
  index partial reuse.

Correction:

- Apply the same valid/padded distinction as MLA:
  - page tables may include the padded tail page;
  - valid sequence length must remain unpadded;
  - any loc/table remap that can expose token rows must map padding rows to
    invalid/sentinel entries or rely on a proven sequence-length mask.
- Current code inspection shows NSA index uses page ids plus explicit valid
  sequence lengths: `GetKAndS` iterates `token_ids < seq_len`, and paged MQA
  logits receive `seqlens_32` / `indexer_seq_lens`.  Therefore tail-page slack
  is not visible if those lengths remain valid-token based.
- The required immediate fix is a regression test that proves index prefetch keeps
  the padded tail page in the dense page table while relying on valid
  `indexer_seq_lens` to exclude slack.  If a future path consumes index page
  tables without an accompanying valid length, it must add an explicit
  sentinel/remap or warning fallback before using the table.
- If index prefetch cannot prove this contract for a shape, emit a warning
  fallback with a narrow reason instead of silently using a risky path.

Tests:

- Add an index partial reuse case with a non-page-aligned suffix:
  - tail page included in dense pages;
  - dense page table has only `ceil(valid_tokens / page_size)` physical pages,
    not cp-size padding;
  - valid sequence length remains the original unpadded token count so the
    index kernels exclude tail slack.

### C6. HiCache metadata conflates logical and physical lengths

Current state:

- `CpHiCacheNodeMetadata.logical_len` is required to be page-aligned.
- `reserve_write_cp()` rejects non-page-aligned `device_indices`.
- `_build_cp_load_back_plan()` and `load_cp()` use `node.host_len` as both the
  logical cache-hit length and physical load length.

Correction:

- Split the metadata into:

  ```text
  valid_len       # radix/scheduler-visible tokens
  padded_len      # physical host/device cache span
  page_owners     # length padded_len / page_size
  owned_positions # positions inside padded_len
  ```

- Scheduler-facing cache hit length must stay `valid_len`.
- Host reservation, backup, load, eviction, and owner-lane capacity must use
  `padded_len`.
- `load_cp()` should validate allocation length against padded physical length,
  while returning enough information for the radix/scheduler path to expose only
  valid tokens.

Tests:

- Metadata with `valid_len=100, padded_len=128, page_size=64`.
- Host hit reports `100` to scheduler but reserves/loads `128` physical slots.
- `page_owners` length is `2`.

Implementation findings recorded before editing:

- `init_load_back()` appends the tensor returned by `load_back()` directly into
  `req.prefix_indices`.  Therefore a CP `load_cp()` implementation must not
  expose padded tail locations through the returned tensor; padded locations may
  be allocated and used for transfer, but the scheduler-visible return must be
  valid-token length only.
- `load_back()` currently slices each reloaded node from a single flat
  `device_indices` tensor with `node.host_len`.  Once `host_len` becomes
  valid-token length, the flat physical allocation offset must advance by
  `metadata.padded_len`, not by `host_len`.
- `load_back()` also increments device-cache residency accounting from the
  tensor returned to the scheduler.  If `load_cp()` returns only valid locs,
  residency accounting must still use the padded physical allocation length so
  GPU cache pressure is not undercounted.
- `_cp_load_back_node_owner_page_counts()` currently rejects non-page-aligned
  device `node.value`.  That is acceptable for the existing page-aligned write
  path, but it is a separate C7/C8 follow-up once radix nodes can store
  valid-length tails backed by padded physical pages.

### C7. Radix page alignment currently floors partial tails

Current state:

- `page_align_keys()` truncates keys to a page boundary.
- `prepare_write_backup_for_req()` calls `page_align_keys()` before probing and
  reserving CP HiCache backup.
- This can preserve correctness, but a short extend can be omitted from host
  cache until it completes a full page.  Repeated short suffixes can therefore
  keep future hits on a fallback/partial path.

Correction:

- Do not append fake tokens to radix keys.
- Keep radix keys valid-token based, but attach physical padded metadata to the
  node/backup record.
- If a radix split would land inside an in-flight padded physical page, either:
  - delay/pending the split; or
  - force the split to a page boundary and keep the valid-tail metadata attached
    to the child.

Tests:

- Write a 100-token suffix and verify the radix-visible key length remains 100
  while physical metadata covers 128 tokens.
- Split a node around the tail page and verify valid/padded lengths remain
  consistent.

Implementation findings recorded before editing:

- `prepare_write_backup_for_req()` floors `req.fill_ids` through
  `page_align_keys()` before reserving CP host backup.  This is the path that
  drops short tails from write-through HiCache.
- `_probe_existing_radix_prefix_len_no_split()` also floors its probe key.  If
  write preparation keeps valid tails, this probe must use the same valid-key
  contract or it can reserve duplicate tail spans.
- `match_prefix()` floors lookup keys before matching.  Even if a valid-tail
  radix node exists, current lookup will not report that tail as a device/host
  hit.
- `_split_node()` delegates CP HiCache metadata split to
  `CpHiCacheNodeMetadata.split()`, which still requires page-boundary splits.
  Supporting arbitrary valid-tail splits is therefore not just a metadata
  change; it needs explicit tail-page ownership semantics or a delayed split.
- `pin_prefix()` also floors keys.  Pinning is not on the hot cache-hit path,
  but it will remain page-granular until the radix valid-tail contract is fully
  audited.
- `HiRadixCache` inherits `RadixCache.cache_finished_req()` and
  `cache_unfinished_req()`.  Both floor keys before `insert()`, so prepared CP
  backups for valid tails would be rolled back as `insert_miss` unless the
  insertion path also keeps valid-tail keys.
- `_key_match_paged()` assumes all keys were page-aligned; when two equal
  partial-tail keys are compared, it can advance by a full `page_size` and
  return a prefix length larger than `min(len(key0), len(key1))`.  Valid-tail
  radix keys require the final compare step to advance only by the remaining
  chunk length.

Implemented C7 slice:

- CP HiCache write preparation now keeps the valid tail instead of flooring
  `fill_ids` before reservation.
- CP `cache_finished_req()` / `cache_unfinished_req()` insertion keeps valid
  tails when `_uses_cp_hicache` is true; non-CP radix behavior still floors.
- CP `match_prefix()` keeps valid-tail lookup keys and reports the valid
  `host_hit_length`.
- CP write reservation pads only to the current tail page and stores
  `metadata.logical_len == valid_len` with `metadata.padded_len` as the physical
  span.
- `_key_match_paged()` now returns the true valid prefix length for a partial
  final page.

Implemented C7 split policy:

- Cache management treats a page as the minimum ownership unit.  If matching an
  already-backed CP HiCache node would split inside a padded physical tail page,
  the match is floored to the previous page boundary instead of splitting that
  physical page.
- Exact valid-tail hits still report the valid length.  The floor policy only
  applies when a shorter/divergent request would require a new radix boundary
  inside the tail page.
- If the floored boundary is zero, the match returns the parent/root node and
  sacrifices the sub-page prefix.  This avoids half-page owner accounting and
  keeps host/device/draft metadata page-granular.
- The same floor policy is applied before forward-time write reservation and
  final radix insertion.  `_probe_existing_radix_prefix_len_no_split()` floors
  backed partial-tail hits so backup reservation starts at the same page boundary
  as `match_prefix()`, and `insert()` floors backed partial-tail splits before
  calling `CpHiCacheNodeMetadata.split()`.

### C8. Owner-lane capacity must be padded-page based end to end

Current state:

- Compute-owner allocation returns one owner per physical page unit, including a
  tail page.
- HiCache capacity planning counts `page_size` tokens per owner page.
- Some entry points still pass scalar valid lengths or page-aligned-only
  `device_indices`.
- `HiRadixCache._cp_load_back_node_owner_page_counts()` still rejects a
  device-resident CP node whose `node.value` is a valid tail shorter than the
  padded physical page span.  This can break load-back owner-lane eviction when
  the victim that should free a lane is a valid-tail node.

Correction:

- Derive required capacity from padded `page_owners`, not valid token counts.
- Admission should use a per-owner vector built from the padded page list.
- Do not introduce an all-reduce/collective to discover capacity agreement; the
  owner pattern should make the required vector deterministic.

Tests:

- A 100-token suffix with owners `[0, 1]` should require one page on owner 0 and
  one page on owner 1.
- Target and draft required vectors must match when draft HiCache is enabled.

Implemented C8 slice:

- CP write admission derives required host capacity from the padded page owner
  vector; a 100-token suffix on 64-token pages requires `(64, 64, 0, ...)`,
  not a 100-token scalar and not CP-size padding.
- Target and draft admission share the same padded required vector; only their
  available-capacity vectors differ.
- Host capacity snapshots count committed and pending valid-tail nodes by
  `metadata.page_owners` / `metadata.padded_len`, so target/draft usage reflects
  physical pages while `host_len` remains valid-token based.
- Load-back owner-lane eviction now pads a device-resident valid-tail victim to
  its physical tail page before counting freed owner pages.  A 6-token value on
  4-token pages frees owner pages `(1, 1, 0, 0)` when it spans logical pages 1
  and 2.

### C9. Draft/EAGLE currently has safe fallbacks, not the final contract

Current state:

- Draft HiCache piggybacks target reservations/positions when
  `draft_host_indices` are present.
- Draft partial-current reuse/prefetch has been disabled or pushed to older
  materialization paths in some cases to avoid EAGLE hangs.
- Existing draft HiCache controller tests cover page-aligned write/load/evict,
  but they do not yet prove that a valid-tail node mirrors the target padded
  physical span while exposing only the valid length.

Correction:

- Draft physical pages must mirror target padded pages.
- Draft valid length must mirror target valid length.
- Draft cannot independently choose a valid-only capacity or prefetch path.
- Re-enable draft/EAGLE partial reuse only after the same page-slot visibility
  mask is proven for draft.

Tests:

- Target/draft reservation with a non-page-aligned suffix uses identical
  `owned_positions` and padded physical length.
- EAGLE one-layer path does not prefetch or consume an unmasked padded tail.

Implemented C9 slice:

- Added target/draft write coverage for a valid-tail suffix.  Target and draft
  reserve the same padded local positions and transfer the same physical device
  span while metadata keeps the valid `logical_len`.
- Added target/draft load coverage for a valid-tail host hit.  Both pools load
  the padded physical page, but `load_cp()` returns only valid logical locs to
  the scheduler.

### C10. Disaggregated transfer has not been audited for valid/padded split

Current state:

- Transfer APIs are page-oriented in several paths, but request metadata remains
  token-length oriented.
- There is no current checklist proving that decode never infers prompt length
  from transferred padded page count.
- Mooncake has CP shared-KV page filtering in
  `MooncakeKVSender.send()` / `filter_kv_pages_for_cp_shared_kv()`, where
  prefill logical pages are filtered by owner lane and remapped to this rank's
  physical page ids while destination pages are selected by request-page
  position.
- NIXL does not have the equivalent CP shared-KV logical-page-position mapping.
  Its sender still slices `req.dst_kv_indices[index_slice]` directly.  Under CP
  shared KV this can silently pair owner-filtered prefill physical pages with the
  wrong decode destination pages.

Correction:

- Transfer page lists may include the padded tail page.
- Request/bootstrap metadata must carry valid token length separately.
- Decode-visible prompt length and max-context checks must use valid length.
- Draft transfer follows target page pattern.
- Until NIXL gets the same logical-position mapping as Mooncake, CP shared-KV PD
  transfer must fail fast for non-Mooncake transfer backends instead of silently
  taking an incompatible path.

Tests:

- Send a non-page-aligned prompt through disaggregation:
  - transferred pages include the padded tail page;
  - decode sees the original valid length;
  - draft and target transfer page counts match when draft KV is enabled.
- Unit-level guard:
  - CP shared-KV page filtering keeps the padded tail page for a valid-tail
    prompt without padding to CP-size pages;
  - `ServerArgs` rejects CP shared-KV prefill with NIXL transfer until that
    backend implements logical-position selection.

Implemented C10 slice:

- Added unit coverage that a 100-token / 64-page prompt covers two transfer
  pages, and owner rank 1 transfers only the tail page selected by logical
  request-page position.  This locks the "pad only to page boundary, not
  cp-size pages" transfer behavior.
- Added a startup validation guard: CP shared-KV prefill disaggregation is
  currently Mooncake-only.  NIXL now fails fast because it still lacks
  Mooncake's logical-page-position mapping for owner-filtered transfer pages.

### C11. Fallback logging must be audited

Current state:

- MLA prefix misalignment now logs a warning fallback.
- Some prefetch misses use debug logs or normal fallback paths.
- Historical silent fallbacks made it hard to identify that async/per-layer or
  page-aligned paths were not actually active.
- The CP shared-KV compute-owner allocation fallback currently logs at INFO and
  does not use the `[CP_SHARED_KV_FALLBACK]` marker.  This is an intended hot
  path fallback from owner-lane/page-aligned allocation to legacy allocation, so
  it must be visible at warning level.

Correction:

- Production-visible fallback from an intended hot path must emit
  `[CP_SHARED_KV_FALLBACK]` or `[CP_HICACHE_FALLBACK]` at warning level with a
  narrow reason.
- Debug-only logs are acceptable only for expected non-fallback states.
- Avoid per-layer spam for expected steady-state events; warn on path changes,
  not every successful layer.

Tests:

- Unit-test representative fallback logs:
  - MLA prefix not page-aligned;
  - index prefetch consume miss after start layer;
  - CP HiCache reservation/load capacity failure.
- Update compute-owner allocation fallback coverage to require warning level and
  the `[CP_SHARED_KV_FALLBACK][compute_owner_alloc]` prefix.

Implemented C11 slice:

- Changed CP shared-KV compute-owner allocation fallback logging from INFO to
  WARNING and added the `[CP_SHARED_KV_FALLBACK][compute_owner_alloc]` marker.
- Tightened the existing unit expectation so repeated fallback events are still
  emitted, but now require the production-visible fallback prefix.

Additional C11 finding:

- Runtime fallback helpers for current reuse, TAI materialize, and TAI fused MLA
  store already log at WARNING, but their messages do not use the standard
  `[CP_SHARED_KV_FALLBACK]` marker.  These helpers cover intended optimized hot
  paths, so they should standardize the marker instead of requiring every caller
  to remember the prefix.

Implemented C11 runtime-helper slice:

- Standardized current-reuse, TAI materialize, and TAI fused MLA store fallback
  helpers to prefix warnings with `[CP_SHARED_KV_FALLBACK]` and a component
  name.
- Added unit expectations for those helper prefixes and tightened the draft
  partial-current-reuse fallback check to require the standard marker.

### C12. Verification gaps before claiming completion

Current state:

- Python syntax checks pass for the edited SGLang files.
- Local pytest collection can be blocked by missing optional dependencies such
  as `pybase64`; this workspace also hit `orjson` during targeted
  `test_nsa_cp_utils.py` collection, then `starlette` after temporary stubbing.
  `test_cp_shared_kv_layout.py` collection also hit `transformers` via
  `sglang.test.test_utils`.
- CUDA kernels must not be tested locally.

Correction:

- Use a temporary local test stub only for optional import blockers such as
  `pybase64` / `orjson`; do not commit stubs.
- If dependency stubbing starts expanding into unrelated web/server packages,
  stop the dependency-stub path and use isolated module loading or
  `py_compile` for the targeted helper.  Do not spend the implementation loop
  repeatedly rediscovering local environment gaps.
- Run CPU/unit tests for extent/split/owner/runtime helpers.
- Run CUDA/TAI kernel verification only on the remote CUDA environment.
- Record remote log evidence for:
  - TAI page-slot compose being used;
  - no warning fallback on the expected hot path;
  - cache-hit request completing without detokenizer hang.

Tests:

- Targeted CPU pytest for:
  - `test_nsa_cp_utils.py`;
  - `test_cp_shared_kv_layout.py`;
  - `test_cp_shared_kv_runtime.py`.
- Remote CUDA benchmark/runtime for the TAI materialize primitive and full ETE
  prefill run.

Verified C12 CPU/unit slice:

- Local syntax verification passed for the touched runtime/prefetch/caller/test
  files with `python -m py_compile`.
- Local targeted pytest remains blocked by the already-recorded local dependency
  gap (`orjson` import via `sglang.srt.utils.common`), so this branch uses the
  remote container as the authoritative unit-test environment for these files.
- Remote container `g0034:/mnt/beegfs/cjy/sglang-dev` passed the three-file CPU
  suite:

  ```text
  PYTHONPATH=python python -m pytest -q \
    test/registered/unit/layers/test_nsa_cp_utils.py \
    test/registered/unit/mem_cache/test_cp_shared_kv_layout.py \
    test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py
  # 123 passed, 5 warnings
  ```

### C13. Remote source sync can expose mixed-version symbols

Current state:

- Remote container targeted new tests passed after scp sync.
- A wider remote run of:

  ```text
  PYTHONPATH=python python -m pytest -q \
    test/registered/unit/layers/test_nsa_cp_utils.py \
    test/registered/unit/mem_cache/test_cp_shared_kv_layout.py \
    test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py
  ```

  produced 102 passes and 19 failures.
- The dominant failure is an import mismatch:

  ```text
  nsa_indexer.py / nsa_backend.py import
  cp_shared_kv_mla_prefetch_should_trace_tiny_extend
  from cp_shared_kv_runtime.py, but the synced runtime file does not export it.
  ```

Correction:

- Treat this as a mixed-version/sync integrity problem before making logic
  changes.
- Compare local and remote `cp_shared_kv_runtime.py`, `nsa_indexer.py`, and
  `nsa_backend.py` symbol expectations.
- Either restore/export the expected helper or sync the matching caller files.
- Re-run the same remote wider suite after the symbol contract is repaired.

Tests:

- Import-level check for the matching caller set:
  `import sglang.srt.layers.attention.nsa.nsa_indexer` and
  `import sglang.srt.layers.attention.nsa_backend`.
- Re-run the three-file remote pytest suite.

Completed C13:

- The stale helper
  `cp_shared_kv_mla_prefetch_should_trace_tiny_extend` is intentionally not a
  runtime API in the current code.  The fix was to sync matching caller files
  (`nsa_indexer.py`, `nsa_backend.py`) with the current runtime instead of
  reintroducing a dead compatibility helper.
- Remote caller import check prints `CALLER_IMPORT_OK`.
- The same remote three-file unit suite now passes (`123 passed, 5 warnings`).

### C14. Wider remote unit suite still exposes non-page-padding regressions

Current state:

- After directly overwriting remote `nsa_indexer.py` and `nsa_backend.py` to
  remove the mixed-version import mismatch, remote py_compile passed.
- The three-file remote unit suite improved to:

  ```text
  114 passed, 7 failed, 5 warnings
  ```

- Remaining failures are:

  1. `test_local_out_cache_loc_logs_every_fallback_event` expects the old
     fallback text `"CP shared KV direct-write fallback"`, while current code
     emits the newer warning prefix
     `[CP_SHARED_KV_FALLBACK][direct_write]`.
  2. MLA and index prefetch stream tests expect prefix materialization to run
     inside the prefetch stream context, but current code records materialize on
     the current stream and only the all-reduce on the prefetch stream.
  3. MLA and index `wait_attention_window()` tests expect the pending event to
     be waited/launched.  Current code returns without waiting in those unit
     setups.
  4. `test_mla_prefetch_min_prefix_pages_uses_cached_token_default_and_can_override`
     expects `SGLANG_CP_SHARED_KV_MLA_PREFETCH_MIN_PREFIX_PAGES=-2` to return
     `16` for `cp_size=4,page_size=64`, but current code returns the dynamic
     token default of `8`.

Correction:

- Treat these as separate from the page-boundary extent work.
- The fallback-log test should be updated to the new warning prefix because C11
  defines `[CP_SHARED_KV_FALLBACK]` warning-level logging as the intended
  production contract.
- Code inspection plus the Phase 8 prefetch doc confirm the current stream/wait
  behavior is intentional: prefix materialize runs on the current stream because
  MLA/index materialize kernels consume SM and should remain ordered; only the
  async all-reduce is enqueued on the prefetch stream.  `wait_attention_window()`
  is a historical hook and must not wait; the real wait remains at next-layer
  `consume(...)` immediately before use.  The stale unit tests should assert this
  deferred-consume model instead of forcing attention-tail waits.
- `SGLANG_CP_SHARED_KV_MLA_PREFETCH_MIN_PREFIX_PAGES=-2` has no separate runtime
  contract in docs or code; all negative values use the same dynamic default:
  `max(cp_size, ceil(_MLA_PREFETCH_DEFAULT_MIN_PREFIX_TOKENS / page_size))`.  The
  expected value changed from 16 to 8 after the default token threshold was reset
  from 1024 to 512; the stale test expectation should be updated, not the code.

Tests:

- Re-run the exact failed tests after each fix, then the three-file remote
  suite.

Completed C14:

- Updated the stale unit expectations to the intended contracts:
  - direct-write fallback logs require
    `[CP_SHARED_KV_FALLBACK][direct_write]`;
  - MLA/index prefix materialize is recorded on the current stream while async
    reduce is recorded on the prefetch stream;
  - `wait_attention_window()` remains a deferred historical hook and does not
    launch/wait pending reduce work;
  - negative
    `SGLANG_CP_SHARED_KV_MLA_PREFETCH_MIN_PREFIX_PAGES` values use the dynamic
    default `max(cp_size, ceil(default_tokens / page_size))`, so
    `cp_size=4,page_size=64` returns `8` with the current 512-token default.
- Remote exact C14 regression set passed:

  ```text
  test_local_out_cache_loc_logs_every_fallback_event
  test_mla_prefetch_materializes_on_current_stream_and_reduces_on_prefetch_stream
  test_index_prefetch_materializes_on_current_stream_and_reduces_on_prefetch_stream
  test_mla_prefetch_attention_window_does_not_launch_pending_reduce
  test_index_prefetch_attention_window_defers_pending_event_wait
  test_mla_prefetch_min_prefix_pages_uses_cached_token_default_and_can_override
  # 6 passed, 3 warnings
  ```
- Remote three-file suite passed after sync:

  ```text
  123 passed, 5 warnings
  ```

## C15: CP valid-tail radix nodes still need page-physical device accounting

Finding:

- Startup can fail immediately after disaggregated prefill warmup with:

  ```text
  token_to_kv_pool_allocator memory leak detected!
  max_total_num_tokens=833024, available_size=833024,
  evictable_size=3, protected_size=0, session_held=0
  ```

- The observed `evictable_size=3` is consistent with EAGLE/bigram warmup:
  a 4-token request is cached as a 3-token bigram radix key.  Under CP HiCache
  the radix key is valid-tail based, while the CP device allocator is
  page-granular.
- `RadixCache.cache_finished_req()` still freed `kv_indices[len(keys):]`.
  For CP + page allocator, freeing the 4th token of a 64-token page releases the
  whole physical page even though the radix node still owns the first 3 valid
  tokens.  The allocator then reports the full pool available while radix still
  reports an evictable 3-token node.
- HiRadix device accounting also still used `len(node.key)` / `len(node.value)`
  in several places.  With valid-tail nodes this is the scheduler-visible valid
  length, not the allocator-visible physical page span.

Correction:

- CP HiCache device residency must account physical page spans while radix
  matching and scheduler-visible prefix lengths remain valid-token based.
- Tail free for CP valid-tail keys must start at the next page boundary, not at
  `len(keys)`, so partial retained pages are never released out from under a
  radix node.
- Splits inside a CP physical page should be floored to the previous page
  boundary (or rejected/deferred at zero) even when the node is not yet backed
  by host metadata.  Page is the minimum ownership unit for both GPU and host
  HiCache.

Tests:

- Add a regression for EAGLE/bigram CP finished-cache tail free: retained
  partial pages must not call allocator free on the in-page tail.
- Add CP page-physical accounting regressions for insert/lock/evict paths.
- Add a split-floor regression for an unbacked CP valid-tail node.

Implemented C15:

- Added `ceil_to_page_len()` and made CP `cache_finished_req()` free suffix
  pages only from the next page boundary after the valid radix key.  This keeps
  a retained valid-tail node from releasing the physical page it still owns.
- Added `HiRadixCache._node_device_resident_len()` and routed CP insert,
  single-node write locks, path locks, load-back locks, regular eviction, and
  backed demotion through allocator-visible physical page lengths.
- Changed the CP split-floor policy to apply to unbacked valid-tail device
  nodes too.  Splits inside a physical page now floor to the previous page
  boundary independent of host-backup state.

Completed C15 tests:

```text
remote g0034 container:
  test_cp_hicache_metadata.py
    90 passed, 5 warnings
  test_cp_hicache_load_back_owner_lanes.py
  test_cp_shared_kv_layout.py
  test_cp_shared_kv_runtime.py
    103 passed, 3 warnings
```

Not-tested C15:

- `test_radix_cache_unit.py` collection currently fails in the remote container
  before running tests because the `sgl_kernel::sgl_per_token_group_quant_8bit`
  operator is not registered in that test import path.  This is an environment /
  test bootstrap issue, not a C15 assertion failure.

## C16: Exact valid-tail hits can make the next backup start mid-page

Finding:

- Current C7/C15 code keeps exact CP valid-tail cache hits token-precise.  That
  means a request can hit, for example, 100 valid tokens on 64-token pages and
  then start its next suffix at token 100.
- `prepare_write_backup_for_req()` derives the backup start as
  `max(req.cache_protected_len, existing_prefix_len)`.  If the protected prefix
  is a valid tail, `start` can be inside a physical page.
- `_cp_required_host_tokens_by_rank()` / `reserve_write_cp()` then call
  `pad_token_locs_to_page_boundary()` on `device_indices[start:]`.  That helper
  is only correct for spans that start at a page boundary; CPU tensors validate
  this, but CUDA tensors intentionally skip the sync-heavy offset check.
- Therefore the production CUDA path can silently build a padded physical span
  from a mid-page start, producing incorrect page-owner accounting and host
  reservation metadata.

Risk:

- This is a correctness risk for repeated prompts where the previous request
  leaves a non-page-aligned valid-tail node and the next request extends beyond
  that exact tail.
- It also conflicts with the current design principle that CP HiCache owns cache
  at page granularity and may sacrifice sub-page cache to keep management simple.

Correction options:

- Preferred under the current page-as-minimum-unit contract: expose only the
  previous page boundary as `cache_protected_len` / `device_indices` for any hit
  that will be extended, so new backup starts at a page boundary.
- More complex alternative: implement explicit partial-page sharing/refcounting
  between a cached prefix tail and the new current suffix.  This is intentionally
  not the current direction.

Tests needed:

- Cache a 100-token valid-tail node, then issue a request that hits those 100
  tokens and extends by another suffix.  Assert write reservation starts at 64,
  not 100, or otherwise fails fast before mid-page padding.
- Add a CUDA/remote regression or source-level guard proving CUDA hot path cannot
  call `pad_token_locs_to_page_boundary()` with a non-page-start span.

Additional finding while implementing C16:

- Flooring only `prepare_write_backup_for_req()` is insufficient.  The same
  exact-valid-tail extension must be treated consistently by radix
  `match_prefix()` and final `insert()`.
- Example with `page_size=64`: an existing 100-token valid-tail node and a new
  120-token request must expose/reuse only the first 64 tokens.  If
  `match_prefix()` exposes 100 or `insert()` later reuses 100, the prepared
  backup from 64..119 either starts mid-page in host accounting or becomes
  unattached/mismatched at insert time.
- Therefore C16 is a three-site contract: prefix probe, scheduler-visible
  prefix match, and insertion duplicate accounting all floor exact valid-tail
  extensions to the previous physical page boundary.

Implemented C16:

- Added a shared exact-valid-tail extension floor helper in `HiRadixCache`.
- `_probe_existing_radix_prefix_len_no_split()` now floors an exact
  non-page-aligned valid-tail hit when the request extends beyond that tail.
- `match_prefix()` now exposes only the floored page-boundary prefix in the same
  situation and splits the radix node at that page boundary.
- `insert()` applies the same floor, stops after the page-boundary split, and
  inserts the recomputed suffix from that boundary so prepared CP backup length
  remains attachable.
- `prepare_write_backup_for_req()` still guards stale scheduler
  `cache_protected_len` by flooring the raw backup start after first checking
  that there is an actual suffix to backup.

Completed C16 tests:

```text
remote g0034 container:
  test_prepare_write_backup_for_req_floors_mid_page_prefix_hit
  test_cp_prepare_probe_floors_exact_valid_tail_when_request_extends
  test_cp_match_prefix_floors_exact_valid_tail_when_request_extends
  test_cp_insert_extends_from_page_boundary_after_exact_valid_tail
```

## C17: Duplicate/no-insert frees are still token-granular under CP pages

Finding:

- C15 fixed the final tail free in `RadixCache.cache_finished_req()` by starting
  CP tail free at the next page boundary.
- The duplicate/no-insert frees remain token-sliced:

  ```text
  cache_finished_req:
    kv_indices[req.cache_protected_len : new_prefix_len]
    kv_indices[req.cache_protected_len : len(keys)]

  cache_unfinished_req:
    kv_indices[req.cache_protected_len : new_prefix_len]
  ```

- With CP page allocators, freeing any loc inside a page can release the whole
  page.  If the free range begins or ends inside a page, it can release a page
  that still contains retained current/request KV or radix-owned valid-tail KV.

Risk:

- Remaining memory-accounting or allocator/radix divergence after the C15 startup
  fix, especially on chunked/unfinished requests or insert-prefix duplicate
  paths.

Correction:

- Audit every CP call to `token_to_kv_pool_allocator.free(kv_indices[a:b])`.
  Under CP page mode, free ranges should either be page-aligned or deliberately
  converted to whole pages that are no longer referenced by radix/request state.
- If a range is not page-aligned and cannot be proven safe, fail fast in tests
  first, then floor/ceil according to ownership semantics.

Tests needed:

- Finished insert where `req.cache_protected_len` and `new_prefix_len` differ
  inside the same page.
- `is_insert=False` finished path with a valid-tail key.
- `cache_unfinished_req()` duplicate-free path with a non-page-aligned protected
  prefix.

Implemented C17:

- Added page-floor/ceil helpers for CP free ranges.
- Routed `cache_finished_req()` duplicate frees, finished no-insert frees, and
  `cache_unfinished_req()` duplicate frees through a CP page-safe range helper.
- Under CP HiCache, duplicate free ranges now skip partial left/right pages and
  only pass full unprotected pages to the page allocator.  The no-insert
  finished path skips the partial left protected-prefix page but still includes
  the finished request's right tail page because no radix node retains it.  This
  prevents a token-granular free from releasing a physical page that still
  contains protected prefix or radix-owned valid-tail KV without leaking
  no-insert tail pages.

Completed C17 tests:

```text
remote g0034 container:
  test_cache_finished_req_cp_insert_duplicate_free_skips_partial_page
  test_cache_finished_req_cp_no_insert_frees_only_full_unprotected_pages
  test_cache_unfinished_req_cp_duplicate_free_skips_partial_page
```

Not-tested C16/C17:

- `test_cp_shared_kv_runtime.py` passed in the remote container.
- `test_cp_shared_kv_layout.py` was attempted in the same remote container, but
  collection aborted inside installed `sgl_kernel` architecture-specific op
  loading before any assertion ran.  The CP HiCache metadata/regression suite
  passed in the same container.

## C18: `cache_protected_len` now mixes valid-token and physical-page semantics

Finding:

- CP valid-tail matching can expose `req.cache_protected_len` as a valid-token
  length.
- Some scheduler/runtime checker paths still assert or compute as if
  `cache_protected_len` is page-aligned physical protection:

  ```text
  scheduler_runtime_checker_mixin.py::_get_batch_uncached_size
  schedule_batch.py::_evict_swa
  ```

Risk:

- Exact valid-tail cache hits may fail assertions or under/over-count uncached
  tokens in memory self-check paths.
- Even when asserts do not fire in the current CP/NSA configuration, this is an
  API ambiguity that can reappear when batch/chunked/SWA paths are combined.

Correction:

- Split request-visible fields conceptually:
  - valid protected length for scheduler/logical prompt progress;
  - physical protected length for allocator/page accounting.
- Until that refactor is done, CP HiCache should floor exposed protected length
  to the nearest page boundary wherever allocator/page accounting consumes it.

Tests needed:

- Valid-tail CP hit through `Req.init_next_round_input()` and scheduler runtime
  memory checker.
- Guard that SWA/page-aligned paths either remain unreachable for CP NSA HiCache
  or receive a page-aligned protected length.

## C19: Prefetcher creation must not gate target partial-current reuse

Finding:

- Remote ETE run `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_001129.log`
  died at `2026-05-29 00:17:24` with:

  ```text
  [CP_SHARED_KV_FAIL_FAST][mla_partial_current_prefetch]
  reason=missing_prefetcher prefix_lens=[320] extend_lens=[10808]
  ```

- The same log shows the async MLA prefetcher was intentionally not created:

  ```text
  create_skip reason=prefix_below_min prefix_pages=5 min_prefix_pages=8
  ```

- A later run showed the inverse shape: `prefix_lens=[16320]` and
  `extend_lens=[16]` passed the prefix gate and created async prefetchers even
  though the current suffix was sub-page.  So "short extend" did not request
  prefetch directly; a large cache-hit prefix made the async prefetch gate pass.

Corrected contract:

- `CpSharedKVMlaPrefetcher` / `CpSharedKVIndexPrefetcher` mean async
  next-layer prefetch only.  If async prefetch is disabled, below threshold, or
  otherwise not worthwhile, do not create a prefetcher object.
- Target MLA partial-current reuse must not depend on prefetcher existence.
  When there is no prefetched prefix handle, `nsa_backend.py` synchronously
  materializes the page-aligned prefix into the slot-dense buffer, all-reduces
  that prefix range, then splices the current KV rows into their suffix page
  slots with tail slack mapped to `-1`.
- Draft/EAGLE cache-hit partial-current reuse remains disabled until the draft
  KV lifetime has its own same-layer contract.  This does not affect target
  partial-current reuse or cache-miss current-only reuse.
- The disabled path is only the old compact materialize/current merge fallback;
  it can expose padded tail slack.  The synchronous page-slot compose path is
  the fallback-free current-layer contract when no async prefetcher exists.

Tests:

- Target cache-hit partial-current with `cp_shared_kv_mla_prefetcher=None` must
  still return true from `should_reuse_current_extend_kv()`.
- `materialize_prefix_and_reuse_current_kv_page_slots()` covers the no-prefetcher
  synchronous compose path and verifies tail slack maps to `-1`.
- Tiny-extend batches with large prefixes should skip async MLA/index prefetcher
  creation before touching pool getters/remap builders.

## C20: Prefetch disabled must not disable full current reuse

Finding:

- Cache-miss/current-only requests (`extend_prefix_lens_cpu=[0]` and
  `seq_lens_cpu == extend_seq_lens_cpu`) do not need prefix materialization or a
  prefetcher: they can reuse the full current forward KV directly.
- Conflating "no prefetcher" with "no current reuse" regresses warmup,
  cache-miss, and tiny-prefix traffic by forcing unnecessary full materialize or
  rebuild work.

Risk:

- Performance regression on no-cache or tiny-prefix traffic.  This is especially
  visible before HiCache warms up or when a prompt falls below the MLA prefetch
  threshold (`prefix_pages < min_prefix_pages`, currently 8 pages by default for
  `cp_size=8,page_size=64`).

Correction:

- Keep `should_reuse_current_extend_kv()` structured as:
  - `current_only` is sufficient for full current reuse;
  - target `partial_current and not current_only` is also sufficient, regardless
    of prefetcher existence, because the backend has synchronous page-slot
    compose;
  - draft/EAGLE `partial_current and not current_only` remains rejected.
- Tests include both target partial-current/no-prefetcher and target
  current-only/no-prefetcher coverage.

## Testing requirements / status

Targeted coverage needed for this contract:

1. `extend_len % page_size != 0` MLA partial current reuse:
   - prefix page-aligned,
   - suffix has page slack,
   - dense buffer includes padded suffix rows,
   - slack locs remap to `-1`.
   - status: page-slot compose is covered; non-prefetch compact fallback is
     guarded by fail-fast source test until a direct forward-path fixture exists.

2. Equivalent NSA index partial reuse case:
   - tail page included in page table,
   - valid current tokens remap,
   - slack positions invalid.

3. HiCache metadata:
   - valid length and padded length differ,
   - `page_owners` length is padded-page count,
   - host hit exposed to scheduler is valid length.

4. Capacity:
   - per-owner required capacity is derived from padded pages;
   - target/draft required counts match.

5. Transfer:
   - page list includes padded tail page;
   - decode-visible sequence length remains valid length.

## Implementation guidance

1. Introduce a small helper/data object for page-aligned cache extents.  Do not
   pass raw `len`, `num_tokens`, or `required_slots` through new code without
   naming whether it is valid or padded.
2. Update MLA prefetch compose first, because it is the observed failing path.
3. Update index partial reuse only after MLA has tests for slack masking.
4. Update HiCache metadata/capacity before enabling this contract for host-hit
   load/backup paths.
5. Any temporary fallback must be a `WARNING` with a narrow reason and should be
   removed after page-padded compose is complete.

## Non-goals for this pass

- Do not change request/logical token length semantics.
- Do not make `split_list` include fake tokens.
- Do not add a hot-path CP collective to reconcile capacity.
- Do not make draft KV manage an independent padded/valid contract.
- Do not silently fall back from the page-aligned contract.

## C21: Large prefix with tiny extend must not enter MLA/index prefetch

Finding:

- The current prefetch creation gate is prefix-sized: it skips MLA/index
  prefetch only when `prefix_pages < min_prefix_pages`.
- The 2026-05-29 remote hang did not violate that gate.  The last batch before
  detokenizer health failure had:
  - `prefix_lens=[16320]` (`255` pages at `page_size=64`), so the prefix gate
    allowed prefetch;
  - `extend_lens=[16]`, less than one page;
  - `has_mla=True has_index=True`, followed by no further forward-layer progress
    and detokenizer timeout.
- Earlier batches with `prefix_lens=[320]` (`5` pages) correctly logged
  `create_skip reason=prefix_below_min ... min_prefix_pages=8` and produced
  `has_mla=False has_index=False`; those are not the observed hang pattern.
- In other words, short extend did not directly request prefetch.  Cache hit
  converted most of the request into a large cached prefix, and the large prefix
  satisfied the prefetch gate while leaving only a sub-page current suffix.

Root-cause analysis:

- MLA/index prefetch is currently a next-layer optimization.  It starts prefix
  materialization/reduce on a separate stream from layer prepare hooks, then the
  consumer waits on the event later.
- Partial-current reuse uses the MLA prefetcher as a page-slot compose contract:
  prefetched prefix rows are reused and the fresh current suffix KV is inserted
  into the padded tail page.
- With `extend_len < page_size`, the suffix page has exactly one CP owner lane
  and many rank-local zero-owned cases.  The current layer is also short enough
  that async prefetch collective ordering and event waits are no longer hidden by
  compute.  A mismatch in next-layer async collective order or event completion
  can present as a CUDA/NCCL wait rather than a Python exception.
- The observed symptom matches this: after `has_mla=True has_index=True` for
  `prefix_lens=[16320], extend_lens=[16]`, no later forward probe appears and
  the server only reports detokenizer heartbeat timeout.

Risk:

- Tiny cache-hit suffixes can pay prefetch launch/materialize/collective overhead
  that cannot be amortized by the short extend.
- More importantly, sub-page suffixes exercise the still-sensitive padded-tail
  partial-current path.  Until that path is proven by ETE, allowing MLA/index
  prefetch on `extend_len < page_size` is a correctness/hang risk.

Correction:

- Do not disable partial-current reuse just because `extend_len` is short.
  Short cache-hit suffixes are exactly where avoiding suffix materialization is
  valuable.
- Add a separate minimum-extend gate only for the async next-layer prefetch
  strategy.  When `extend_len` is below the async threshold, do **not** create an
  MLA/index prefetcher object; a prefetcher object now means async next-layer
  prefetch machinery exists.
- Target partial-current reuse is independent of prefetcher existence.  Without
  a prefetched prefix handle, the backend takes the synchronous page-slot compose
  path: materialize/reduce the page-aligned prefix in the current layer and then
  splice current KV rows into their padded suffix page slots.
- Full current reuse is also independent of prefetcher existence.  Cache-miss
  current-only batches keep using the forward KV directly even when async
  prefetch is disabled, skipped by threshold, or not created.
- If partial-current reuse is intentionally rejected, for example the current
  conservative draft/EAGLE cache-hit guard, that rejection must not disable
  current-only/full-current reuse for other request shapes.
- C21 initially defaulted the async gate to one page when `page_size` is known;
  C22 supersedes that default with one page per CP lane after the 65-token
  repeated-request hang showed that one page was still too small.  The env
  override remains available for experiments.
- This must not force tiny cache-hit suffixes back to compact full
  materialization.

Tests:

- Add runtime helper coverage for the min-extend threshold default and override.
- Add MLA/index prefetch creation tests where `prefix_pages >= 8` but
  `extend_len < page_size`; both prefetchers should return `None` before
  touching pool getters/remap builders, while current reuse remains available.
- Add a synchronous no-prefetch compose test proving sub-page current suffix rows
  are filled while tail slack in the current page is masked.

Verified:

- Remote container `/sgl-workspace/sglang-tai` on `g0034`:
  - `test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py`: `73 passed`
  - `test/registered/unit/mem_cache/test_cp_hicache_metadata.py`: `90 passed`

## C22: One-page async prefetch gate still admits repeated-request hangs

Finding:

- Latest remote log `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_134616.log`
  shows the first 40k request completed and backed up CP HiCache node 4.
- The repeated request hit the cached prefix and created both MLA and index
  prefetchers on every CP rank:
  `prefix_lens=[40320]`, `extend_lens=[65]`, `prefix_pages=630`,
  `min_async_extend_tokens=64`, `has_mla=True has_index=True`.
- No `index_start_request` or `forward_probe` appears after those `create_result`
  lines; the next lines are detokenizer heartbeat timeout and shutdown cleanup.
- Therefore C21's one-page default threshold is insufficient.  It blocks
  sub-page suffixes but still allows a 65-token suffix, i.e. just over one page,
  to enter async next-layer prefetch.

Correction:

- The intended production default is CP-lane scale, not one page: with
  `cp_size=8,page_size=64`, async MLA/index prefetch should require at least
  `8 * 64 = 512` current extend tokens unless explicitly overridden.
- This matches the owner-lane mental model: below one page per CP rank, the
  prefetch collective and page-slot compose overhead is not amortized and is a
  correctness risk while padded-tail ETE coverage is still incomplete.
- This gate only controls async prefetcher object creation.  Target
  partial-current reuse and full current reuse must continue through the
  synchronous page-slot compose/current-splice path when no prefetcher exists.

Required test update:

- Change the min-extend helper contract from "one page" to "one page per CP
  lane" when both `cp_size` and `page_size` are known.
- Add a regression case for `extend_len=65,page_size=64,cp_size=8` proving both
  MLA and index prefetchers return `None` before touching pool getters/remap
  builders.

Verified C22:

- Remote container `/sgl-workspace/sglang-tai` on `g0034`:
  - `python -m py_compile` for `cp_shared_kv_runtime.py`,
    `cp_shared_kv_prefetch.py`, and `test_cp_shared_kv_runtime.py` passed.
  - Targeted C22 tests passed: `2 passed, 2 subtests passed`.
  - Full `test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py` passed:
    `73 passed, 2 subtests passed`.
- Local pytest remains blocked by missing optional deps such as `orjson`; remote
  container is the authoritative verification environment for this path.

## C23: Repeated EAGLE/bigram requests can expose 65-token tiny suffixes on 64-token pages

Finding:

- A repeated request can still show `extend_lens=[65]` even when the logical
  prompt is identical and page flooring is only supposed to lose at most
  `page_size - 1` tokens.
- The extra token comes from EAGLE/bigram key semantics plus SGLang's existing
  one-token prefix-cache holdback:
  - request sequence length in the observed log: `seq_len=40385`;
  - scheduler prefix match caps token keys at `input_len - 1`;
  - EAGLE radix keys are bigrams, so `convert_to_bigram_key()` reduces that key
    length by one more token;
  - effective matchable key length becomes `40385 - 2 = 40383`;
  - page-flooring to 64-token pages gives `40320`;
  - forward therefore sees `40385 - 40320 = 65` extend tokens.

Implication:

- For non-EAGLE page-granular reuse, the repeated-request suffix can be up to
  `1 + (page_size - 1)` tokens because the last token is held back.
- For EAGLE/bigram reuse, it can be up to `2 + (page_size - 1)` tokens.  With
  `page_size=64`, the maximum tiny suffix is therefore `65`, matching the
  observed hang log.
- This does not mean the user prompt changed; it is an artifact of the current
  cache key contract.

## C24: Tiny page-count cache-hit suffixes still enter NSA in-seq CP split

Finding:

- Latest remote log `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_140440.log`
  shows C22 is active: the repeated cache-hit request with
  `prefix_lens=[40320]`, `extend_lens=[65]`, `page_size=64`, `cp_size=8`
  skips both MLA and index async prefetchers:
  `create_skip reason=extend_below_min ... min_async_extend_tokens=512`,
  followed by `create_result ... has_mla=False has_index=False`.
- The same request then hangs before any `forward_probe` / `forward_layer`
  line for layer 2; the next relevant lines are detokenizer heartbeat timeout.
- The earlier log `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_134616.log`
  showed the same 65-token repeated suffix hanging when async prefetchers were
  created.  Therefore async prefetch creation is not the root cause.
- `can_cp_split()` still allows NSA in-seq CP split whenever
  `sum(extend_seq_lens_cpu) >= cp_size`.  For the observed repeated request,
  page-aligned physical coverage is only `ceil(65 / 64) = 2` pages, far below
  `cp_size=8`.
- `build_page_aligned_in_seq_split_list(total_len=128, extend_len=65,
  cp_size=8)` produces a degenerate split where only two page units are real and
  most CP lanes get zero valid suffix tokens.  This is the common condition
  across both hang logs.

Correction:

- Async prefetch gating alone is insufficient.  CP shared-KV page-aligned
  in-seq split itself should require at least one physical suffix page per CP
  rank: `ceil(extend_len / page_size) >= cp_size`.
- Below that threshold, run the request without NSA in-seq CP split rather than
  generating mostly-zero CP segments.  This keeps the cache page contract
  intact while avoiding the fragile zero-lane distributed path.
- This must not disable target current reuse.  With no async prefetcher, the
  backend should still use synchronous prefix materialize + current splice for
  cache-hit suffixes, and full current reuse for cache-miss/current-only
  requests.

Implementation:

- `can_cp_split()` now skips CP shared-KV in-seq split when a page-aligned
  cache-hit suffix has fewer padded suffix pages than CP lanes.  For example
  `extend_len=65,page_size=64,cp_size=8` has two physical suffix pages and no
  longer creates mostly-zero CP segments.
- `can_reuse_current_extend_kv()` now treats tail-padded `out_cache_loc` as
  eligible when it still contains all valid current suffix rows.  The backend
  slices partial-current `current_kv_cache/current_locs` down to the valid
  suffix length before synchronous prefix compose, so padding rows do not become
  current page-slot data.
- Current-only/full-current reuse remains unsliced in the existing path because
  padded current-only CP requests already rely on the padded query/cache shape.

Verified C24:

- Remote container `/sgl-workspace/sglang-tai` on `g0034`:
  - `python -m py_compile` for `utils.py`, `cp_shared_kv_runtime.py`, and
    `nsa_backend.py` passed.
  - Red tests first failed on the old code:
    `test_can_cp_split_skips_cp_when_radix_hit_suffix_has_too_few_pages`,
    `test_can_cp_split_skips_cp_when_page_units_do_not_cover_all_lanes`, and
    padded-current-reuse coverage.
  - After the fix, the targeted C24 tests passed: `4 passed`.
  - Related unit suite passed:
    `test_nsa_cp_utils.py`, `test_cp_shared_kv_runtime.py`,
    `test_cp_shared_kv_layout.py` => `128 passed, 5 warnings, 2 subtests passed`.

## C25: 2026-05-29 remote "hang" snapshot is idle prefill plus L1 owner-lane eviction churn

Finding:

- Latest remote log `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_143924.log`
  did not show a prefill scheduler deadlock at the time of inspection:
  - `/health` responded `200 OK`;
  - metrics reported `num_running_reqs=0`, `num_queue_reqs=0`,
    `num_prefill_prealloc_queue_reqs=0`;
  - detokenizer `py-spy` stack was idle in `recv_pyobj`;
  - scheduler ranks were in the normal idle receive path:
    CP0 sampling at `recv_requests()` nonblocking ZMQ receive and CP1-7
    sampling at `broadcast_pyobj()` from `recv_requests()`.
- The last completed user POST in that log was at `2026-05-29 14:52:08`.
  After that, the log only showed periodic health/internal tiny one-token
  prefill batches, so the prefill process itself had no visible user request
  pending.
- The performance collapse immediately before that idle period coincided with
  repeated L1/device owner-lane eviction and load-back churn, not host-L2
  eviction:
  - no `deterministic CP host eviction before write`,
    `_evict_host_for_physical_slots`, or host-full fallback appeared in the
    log after startup;
  - `HiCache-load` repeatedly had low per-owner availability and evicted device
    victims before CP load-back, e.g. node `873` loaded `101632` host tokens
    but first evicted victims `[627,604]` for `102016` device tokens;
  - generic device eviction then also evicted large backed nodes for much
    smaller capacity deficits, e.g. `num_tokens=10752` evicted `79488`, and
    `num_tokens=19968` evicted `91200`.

Implication:

- The current evidence points to L1/device cache thrashing under CP owner-lane
  constraints once the working set is saturated.  Host cache may be nearly full
  externally, but this log's hot path is not host eviction; it is repeated
  host-hit load-back requiring device-owner-lane free pages, followed by
  coarse whole-node device eviction.
- The existing CP load-back owner-lane planner avoids random lane mismatches,
  but its eviction granularity is still whole radix nodes.  When leaves are
  large, a small lane deficit can evict a much larger cached prefix, destroying
  future L1 hits and increasing H2D load-back work.
- The generic compute-owner allocation eviction path still calls
  `tree_cache.evict(EvictParams(... owner_lane_deficits=...))`, but
  `owner_lane_deficits` is not consumed by `HiRadixCache.evict()`.  It therefore
  remains a coarse total-token eviction path and can over-evict when only a few
  owner lanes are short.

Next correction target:

- Do not treat this snapshot as an async-prefetch hang.  The next fix should
  reduce saturated-L1 owner-lane churn:
  1. add an owner-lane-aware device eviction path for compute-owner allocation,
     reusing the CP load-back victim selection logic where possible;
  2. cap or skip load-back when the required owner-lane eviction cost is far
     larger than the host-hit benefit, unless the cache-hit prefix is large
     enough to amortize the churn;
  3. longer term, split/schedule radix leaves at page-aligned boundaries before
     async backup/load so eviction can free page-sized or small-node units
     instead of whole long-request leaves.

## C26: 2026-05-29 hang is a scheduler collective-order divergence, not only idle L1 churn

Finding:

- A later remote stack sample on `g0034` for
  `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_143924.log` showed a real
  distributed scheduler hang:
  - CP0 / CP6 were in `recv_requests()->broadcast_pyobj()`;
  - CP1 / CP2 / CP5 / CP7 were in
    `scheduler_dp_attn_mixin.all_gather()` from
    `prepare_mlp_sync_batch_raw()->maybe_prepare_mlp_sync_batch()`;
  - CP3 was in idle memory check and CP4 in the prefill event-loop boundary;
  - detokenizer was idle in `recv_pyobj`.
- This means ranks disagreed on the next collective: some entered the DP/attn
  MLP-sync `all_gather`, while others entered scheduler request broadcast.
  That is a collective-order divergence and can deadlock even when queues and
  metrics look empty.

Immediate debugging target:

- Read `event_loop_normal_disagg_prefill`,
  `get_next_disagg_prefill_batch_to_run`, and
  `scheduler_dp_attn_mixin.maybe_prepare_mlp_sync_batch` to identify how one CP
  rank can decide it has a batch requiring MLP sync while another decides it is
  idle and returns to `recv_requests`.
- The earlier owner-lane eviction churn remains a performance issue, but it is
  not sufficient to explain this collective-order mismatch.

Correction C26:

- Added a single-DP idle fast path in `prepare_mlp_sync_batch_raw`: when
  `local_batch is None`, `dp_size == 1`, and `SGLANG_SCHEDULER_SKIP_ALL_GATHER`
  is not enabled, return `None` without building an idle batch or entering the
  MLP-sync all-gather.
- Rationale: CP in-seq split runs inside one DP scheduling domain in the current
  deployment.  All TP/CP ranks should have identical batch presence.  An idle
  all-gather only exchanges zero metadata; after tiny internal health requests
  it can overlap with the next request broadcast and produce the observed
  split: some ranks in Gloo `broadcast_pyobj`, others in NCCL/CPU MLP-sync
  `all_gather`.
- This does not remove MLP sync for real batches or for multi-DP cases where a
  rank may need an idle batch because another DP worker has work.

## C27: 2026-05-29 request failure evidence is not in the prefill log

Finding:

- In the stopped run using `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_152501.log`,
  the prefill log does not contain the request-failure traceback:
  - no `Traceback`, `RuntimeError`, scheduler-dead, health-failure, OOM, or
    Python exception lines after startup;
  - HTTP status counts in the prefill log are all `200`;
  - the last visible prefill log line is a successful health check at
    `2026-05-29 15:45:43`.
- All prefill/decode Python workers were already stopped/zombie when inspected.
  Decode on `g0035`/`g0036` had no persisted stdout/stderr for this run; the
  process output was attached to a TTY rather than a log file.  Therefore the
  concrete decode/router-side request failure cannot be reconstructed from the
  available files.
- Decode workers on both decode hosts show zombie creation around
  `2026-05-29 15:27:26-15:27:39`, while the prefill log continued accepting
  successful `/v1/chat/completions` requests until `15:43:39` and health checks
  until `15:45:43`.  This strongly suggests the user-visible request failure is
  downstream of decode/router availability or external process stop, not a
  prefill-side HTTP 4xx/5xx captured in the HiCache log.

Implication:

- Do not infer a CP HiCache correctness failure from this prefill log alone.
  The available evidence is an observability gap: decode/router failure output
  was not persisted.
- For the next ETE run, decode/router stdout must be persisted with `tee` or an
  equivalent log sink; otherwise a stopped decode process leaves only zombie
  metadata and no Python traceback.

Next correction target:

- Continue code fixes that are already evidenced in prefill logs (owner-lane
  device eviction churn and excessive tiny/internal-request CP fallback logging),
  but treat this specific request failure as unrooted until decode/router logs
  are captured.

C25 correction update:

- Implemented owner-lane-aware `HiRadixCache.evict()` handling when
  `EvictParams.owner_lane_deficits` is set.  The eviction path now plans device
  victims using owner-lane contribution, so a low-priority non-contributing
  leaf is no longer evicted before a higher-priority leaf that actually frees
  the deficit lane.
- Reduced `_evict_for_compute_owner_lanes()` requested eviction from
  `deficit_pages * page_size * cp_size` to `deficit_pages * page_size` because
  the deficit vector is already counted in owner-lane pages.  This removes the
  previous full-CP-stripe over-eviction for a single-lane deficit.
- Remote verification on `g0034` container:
  - RED before fix:
    `test_owner_lane_evict_params_choose_deficit_contributing_victim` evicted
    the non-contributing `[8,9,10,11]` page instead of the deficit owner page.
  - GREEN after fix:
    targeted test passed.
  - Related suite passed:
    `test_cp_hicache_load_back_owner_lanes.py` => `9 passed`.

## C28: 2026-05-29 accept length collapse means EAGLE draft rejection, not a HiCache hit-rate counter

Finding:

- Decode `accept len` is reported from `spec_num_accepted_tokens / spec_num_forward_ct`.
  `update_spec_metrics()` adds `num_accepted_tokens + batch_size`, while
  `_resolve_spec_overlap_token_ids()` defines `num_accepted_tokens` as
  `sum(accept_lens) - len(batch.reqs)`.  Therefore a logged accept length near
  `1.0` means EAGLE accepted zero draft tokens and only produced the mandatory
  target token per verify step.
- The current remote decode processes are running with EAGLE, but not with
  `SGLANG_CP_DRAFT_SHARED_KV=1` in their environment.  Prefill has
  `SGLANG_CP_DRAFT_SHARED_KV=1`, CP shared KV, HiCache, current reuse, and MLA
  prefetch enabled.  Decode still includes draft KV buffers through its draft
  worker, so this is not alone proof of missing transfer, but it is an important
  deployment asymmetry to keep visible.
- Current live metrics on the new run already show low/uneven speculative
  acceptance on some decode DP ranks (for example `spec_accept_length` around
  `1.05` on DP4/DP9 and higher on DP8/DP13), while several ranks are still at
  zero because they have not reported a recent decode window.

Implication:

- The accept-length symptom should be debugged as target/draft state divergence
  or sampling/draft-quality degradation on decode, not as a direct HiCache host
  capacity issue.  The key invariant to verify is: after disaggregated prefill
  and any cache-hit/current-reuse/load-back path, target KV, draft KV, sequence
  lengths, and draft hidden state used by EAGLE are still aligned for the same
  request positions.
- Next evidence should be low-frequency decode-side EAGLE acceptance diagnostics
  keyed by request/cache-hit status and whether draft KV buffers were received;
  per-layer prefill logs are not sufficient to prove why decode rejects drafts.

### C29. 2026-05-30 live run evidence: accept-len collapse is decode-side EAGLE rejection, not missing CP current reuse

Evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_160555.log`, `decode0_20260529_160608.log`, and `decode1_20260529_160611.log`:

- Decode-side `accept len: 1.00` recurs across both decode nodes. Metrics at 2026-05-30 00:16 CST showed examples: DP4=1.0 on g0035 and DP9=1.0 on g0036. This means speculative draft tokens are rejected; the accepted token is only the mandatory target token.
- The low-accept events are not limited to requests with active disaggregation transfer. Log parsing showed many `accept len <= 1.001` events with `#transfer-req: 0`, so the symptom persists after transfer/preallocation, not only during the transfer window.
- Prefill CP shared KV current reuse is active in the same run. Counts: `forward_partial_current_reuse=2032`, with `used_prefetch=True=1104` and `used_prefetch=False=928`. Small-prefix cases below the async-prefetch threshold still use synchronous current compose, e.g. `reason=missing_prefetcher prefix_lens=[320] ... forward_partial_current_reuse used_prefetch=False`.
- Therefore `prefix_below_min` disables async prefix prefetcher creation only; it does not disable partial/full current reuse. It should not by itself explain accept length collapsing to 1.
- Decode startup remains asymmetric: g0035/node-rank0 command lacks `--speculative-attention-mode prefill`, while g0036/node-rank1 includes it. Both decode launch environments only expose `SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1` and `SGLANG_DISAGGREGATION_QUEUE_SIZE=8` among CP/disagg flags.

Current hypothesis to test next: accept-len collapse comes from decode-side EAGLE draft/target state divergence or decode launch/config asymmetry, not from CP shared KV partial current reuse being disabled. Add low-frequency decode-side EAGLE diagnostics around accept-lens / draft KV availability / transferred request state before changing prefill cache logic again.

C29 correction note:

- `--speculative-attention-mode prefill` is the default in `ServerArgs`; therefore g0035 omitting the explicit flag and g0036 spelling it out is not enough to prove a runtime mismatch. Keep checking actual parsed server args if needed, but do not treat the command-line spelling difference alone as root cause.

### C30. 2026-05-30 router failures are downstream of prefill one-page cache accounting failure

Evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_160555.log` and router errors around `2026-05-29 16:19:08`:

- Router `Prefill server failed (CRITICAL)` errors align with the prefill scheduler exiting, not with an independent router/decode routing bug.
- The direct prefill failure is the idle memory checker:
  `token_to_kv_pool_allocator memory leak detected! self.max_total_num_tokens=833024, available_size=16384, evictable_size=816576, protected_size=0, session_held=0`.
- The accounting gap is exactly one page: `833024 - (16384 + 816576 + 0 + 0) = 64` tokens, matching the configured page size.
- Immediately before the failure, logs show tiny internal/prefill batches where scheduler allocation is page-granular (`#new-token: 64`) but CP shared KV metadata has valid lengths such as `seq_lens=[1]` or `seq_lens=[6]`, `real_pages=1`.
- Therefore the next root-cause target is a one-page lifecycle/accounting mismatch around tiny CP+EAGLE disagg prefill requests, especially the path `cache_unfinished_req(req)` before KV transfer and `release_kv_cache(req, tree_cache)` after transfer completion.

Implication:

- The router error is a symptom: once prefill rank workers die, decode/router cannot fetch prefill KV and reports failures.
- This is separate from the low EAGLE `accept len` symptom. Low accept length remains a decode-side draft rejection/state divergence issue; the one-page accounting failure explains prefill death and router errors.

Current hypothesis to verify before fixing:

- Disaggregated prefill inserts and locks an unfinished radix node before transfer. On transfer completion, the generic `release_kv_cache()` path calls `cache_finished_req()` again, which can re-insert/split a finished-key view. For tiny valid lengths under page granularity, this can leave exactly one physical page outside `available + evictable + protected` accounting.

### C31. Repeated tiny EAGLE requests leak one CP HiCache page when exact matched tail is not page aligned

New root-cause evidence:

- A minimal remote repro with CP HiCache, `page_size=64`, EAGLE/bigram keys, and repeated `seq_len=6` requests shows the first request is balanced after release, but every repeated exact hit loses one 64-token page:
  - after first release: `avail=8128 ev=64 prot=0 sum=8192`;
  - after second release: `avail=8064 ev=64 prot=0 sum=8128`;
  - after third release: `avail=8000 ev=64 prot=0 sum=8064`.
- This matches the live crash where `max_total_num_tokens - (available + evictable + protected + session_held) = 64`.
- Cause: CP HiCache uses valid-token radix keys but page-granular allocator accounting. For repeated tiny EAGLE requests, the full key is an exact non-page-aligned hit, e.g. `len(keys)=5` for `seq_len=6`.
- `cache_unfinished_req()` / `cache_finished_req()` call `_free_kv_indices_range(kv_indices, cache_protected_len, new_prefix_len)` for duplicate prefixes. In CP mode `_free_kv_indices_range` ceil-aligns `start` and floor-aligns `end` unless `include_partial_tail=True`. With `start=0,end=5,page_size=64`, the duplicate range floors to empty, so the newly allocated physical page is neither freed nor represented by a new radix node. The existing cached node accounts only its own page, leaving the new duplicate page leaked.

Correction target:

- When an insertion result is an exact full-key duplicate (`new_prefix_len == len(keys)`) under CP HiCache, free the duplicate request KV range with `include_partial_tail=True`. This frees the newly allocated physical tail page while preserving the existing cached node. Keep normal partial-split/new-node behavior page-floored to avoid freeing pages retained by radix nodes.

C31 correction:

- `RadixCache.cache_finished_req()` and `cache_unfinished_req()` now free duplicate KV ranges with `include_partial_tail=True` when CP HiCache insertion reports an exact full-key duplicate (`new_prefix_len == len(keys)`).
- This is intentionally limited to exact duplicate hits. Partial/new-node insertions still use the page-floored free path so a newly inserted radix tail page is not accidentally returned to the allocator.
- Added regression coverage for repeated tiny EAGLE exact hits: three repeated `seq_len=2` requests must keep `available + evictable + protected == allocator.size` after each release.

Verified C31:

- Local compile: `python -m py_compile python/sglang/srt/mem_cache/radix_cache.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py` passed.
- Remote `g0034` container `/sgl-workspace/sglang-tai`:
  - targeted regression passed: `test_repeated_tiny_eagle_exact_hit_frees_duplicate_tail_page`;
  - related owner-lane/CP HiCache suite passed: `test_cp_hicache_load_back_owner_lanes.py => 10 passed`.

### C32. Decode accept-length collapse needs draft-transfer evidence, not more prefill cache changes

Finding:

- The latest persisted decode logs (`decode0_20260529_160608.log`,
  `decode1_20260529_160611.log`) show low EAGLE acceptance after transfer has
  already drained: parsing 970 decode-stat lines gives 855 lines with
  `#transfer-req: 0`, average `accept len=1.342`, and 134 exact `accept len=1.00`
  windows.  Therefore the symptom is not limited to the prefill-to-decode
  transfer polling window.
- Prefill did attach the CP draft HiCache mirror in the same run.  All eight CP
  ranks logged `[HiCache-draft] attached CP draft KV host pool`, and the host
  budget was shared across target+draft.  This rules out the simple hypothesis
  that prefill had no draft host mirror.
- The logs do not prove whether the disaggregation transfer actually registered
  and copied draft KV/state buffers, because `SGLANG_CP_DRAFT_SHARED_KV_DEBUG`
  was not enabled; `draft_kv_buffer_count`, `draft_state_buffer_count`, and
  per-room draft transfer records were therefore absent.
- Decode `accept len ~= 1` still means EAGLE draft rejection: the decode metric
  adds the mandatory target token, so `1.00` is zero accepted draft tokens.

Implication:

- Do not continue changing CP partial current reuse or HiCache load-back based
  solely on this accept-length symptom.  The next evidence must come from
  decode-side EAGLE state and disaggregation draft-transfer metadata:
  transferred draft KV/state buffer counts, metadata hidden/topk presence, and
  low-frequency per-request accept-lens after transfer.
- A safe next step is gated diagnostics, not an unconditional behavior change:
  log startup/transfer draft-buffer availability once per rank and optionally
  sample low-accept decode batches under an explicit debug env.

C32 correction:

- Added gated decode-side EAGLE acceptance diagnostics controlled by
  `SGLANG_EAGLE_ACCEPT_DEBUG=1` and sampled by
  `SGLANG_EAGLE_ACCEPT_DEBUG_INTERVAL` (default 128).  The log is warning-level
  only when a sampled batch has zero accepted draft tokens; otherwise it is
  interval-sampled info.  It records per-request accepted draft count, prompt
  length, output length, cached token count, PD metadata tensor presence/shape,
  and current `spec_info` topk/hidden shapes.
- This is diagnostic-only.  It does not change EAGLE sampling, KV transfer,
  cache load-back, or CP shared KV behavior.
- Synced `environ.py` and `scheduler_output_processor_mixin.py` to remote
  `g0034:/mnt/beegfs/cjy/sglang-dev`; remote container py_compile passed.

C32 correction update:

- Extended `SGLANG_EAGLE_ACCEPT_DEBUG=1` to print one startup KV-manager audit per
  rank on both prefill and decode.  The audit includes target/draft KV buffer
  counts and target/draft state buffer counts.  This avoids enabling the much
  noisier `SGLANG_CP_DRAFT_SHARED_KV_DEBUG` just to answer whether draft KV/state
  transfer surfaces are registered.
- Synced `prefill.py` and `decode.py` to remote `g0034:/mnt/beegfs/cjy/sglang-dev`;
  remote container py_compile passed for the updated files.

### C33. EAGLE accept startup diagnostic must read speculative config from scheduler-owned state

Finding:

- Enabling `SGLANG_EAGLE_ACCEPT_DEBUG=1` caused prefill startup to fail in `PrefillBootstrapQueue._init_kv_manager()` with `AttributeError: 'PrefillBootstrapQueue' object has no attribute 'spec_algorithm'`.
- The diagnostic added in C32 incorrectly assumed the bootstrap/prealloc queue objects expose `self.spec_algorithm`. They only retain `self.scheduler`; the speculative algorithm lives on `self.scheduler.spec_algorithm` in these queue classes.
- The same incorrect diagnostic reference existed in `DecodePreallocQueue._init_kv_manager()`, which would have produced the analogous decode startup failure under the same debug env.

Correction:

- Read `self.scheduler.spec_algorithm.is_eagle()` in both prefill and decode KV-manager startup audits.
- This is a diagnostic-path-only fix. It does not change KV registration, cache ownership, transfer, prefetch, or EAGLE acceptance behavior.

Guardrail:

- Future diagnostics inside disaggregation helper queues should use scheduler-owned runtime state unless the queue constructor explicitly copies that state onto `self`.

### C34. 2026-05-30 EAGLE accept debug run: partial accept collapse correlates with cache-hit decode, not missing draft-buffer registration

Evidence from live logs under `/mnt/beegfs/cjy/log/` after enabling `SGLANG_EAGLE_ACCEPT_DEBUG=1` on prefill and both decode nodes:

- Active files inspected:
  - prefill: `sglang_cp_hicache_20260529_173736.log`
  - decode node0: `decode0_20260529_174040.log`
  - decode node1: `decode1_20260529_174053.log`
- Processes were still alive at 2026-05-30 01:53 CST. No new traceback/health-failure was found in the inspected tail.
- Startup audit now proves the simple “draft buffers are not registered” hypothesis is false for this run: target KV buffers, draft KV buffers, target state buffers, and draft state buffers are all visible in the disaggregation KV manager.
- Decode-side EAGLE debug parsing found `25009` logged samples and `876` decode batch stat lines. Logged samples are intentionally biased toward zero because every zero-accepted sample is logged, while non-zero samples are interval-sampled.
- Even accounting for that bias, the low-accept symptom is real in aggregate decode stats: decode batch `accept len` min `1.00`, average `1.412`, median `1.27`, and `168/876` windows at `<=1.05`.
- Low accept is not only a transfer-window effect: `#transfer-req: 0` has `141/765` windows at `accept len <= 1.05`; `#transfer-req: 1` has `22/93` such windows.
- Logged zero-accepted samples appear across cache-hit sizes, not only tiny prefix: top examples include cached `320`, `13312`, `14208`, `15680`, `62528`, `67968`, `74496`, `75520`, `88256`, and `98112`.
- Prefill in the same run heavily exercised CP current reuse: `forward_partial_current_reuse=2464`, with `used_prefetch=False=1432` and `used_prefetch=True=1032`. For small prefix `320` (`5` pages, below `min_prefix_pages=8`), async prefetcher creation was skipped but target partial-current sync compose still ran.
- Draft partial current reuse remained disabled and logged fallback for the small-prefix EAGLE/NextN path, e.g. `reason=draft_partial_current_reuse_disabled ... prefix_lens=[320] extend_lens=[10808]`.

Implication:

- The current root-cause target is narrower than “EAGLE disabled” or “draft KV buffers missing”. The suspicious path is cache-hit prefill with CP shared KV current reuse where target uses partial-current reuse but draft uses full materialize/fallback, followed by decode-side draft rejection.
- This still does not prove that partial-current reuse is the root cause: logged zero-accept samples also appear on large cached prefixes with async prefetch enabled. The next check must trace how prefill target hidden/topk and draft hidden/KV are captured and transferred to decode, especially when cached tokens are non-zero.
- Do not add more cache or prefetch behavior changes solely from accept-len logs. First inspect EAGLE prefill capture and disaggregation transfer metadata paths for target/draft position alignment.

### C35. 2026-05-30 new EAGLE accept debug log: cache-hit runs still reject draft tokens after transfer drains

Evidence from the current live run:

- Active files:
  - prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_173736.log`
  - decode node0: `/mnt/beegfs/cjy/log/decode0_20260529_174040.log`
  - decode node1: `/mnt/beegfs/cjy/log/decode1_20260529_174053.log`
- As of the first inspection, prefill and both decode launch processes were still alive and grep found no traceback, health failure, scheduler fatal error, or memory-leak exception in these three logs.
- Decode-side parsing across both decode logs found `3707` aggregate `Decode batch` windows with average accept length `1.400`, median `1.30`, min `1.00`, and `745` windows at `<=1.05`. This confirms the accept-length symptom remains real in the new run.
- The issue is not only an active transfer-window artifact: `#transfer-req: 0` windows still averaged `1.394` and had `609/3005` windows at `<=1.05`.
- Cache-hit remains the strongest visible correlate: windows classified near decode EAGLE debug samples had `cached0 n=26 avg=2.21 <=1.05=0`, but `cached_gt0 n=3681 avg=1.394 <=1.05=745`. Cached `320` windows averaged `1.484`, while larger cached prefixes still averaged around `1.38`.
- The collapse is not only tiny-extend: inferred `origin-cached` buckets all remain low (`1..63` avg `1.422`, `64..127` avg `1.314`, `2048..8191` avg `1.266`, `8192+` avg `1.430`). Therefore treating only sub-page or sub-CP-page suffixes is insufficient.
- Prefill CP shared-KV logs show current reuse is active in the same run: `forward_partial_current_reuse=8864`, `has_prefetcher=True=2608`, `has_prefetcher=False=6440`, `prefix_below_min=2864`. Small-prefix cases below the async prefetch threshold still use synchronous current compose.
- Explicit fallback counts are narrow: `CP_SHARED_KV_FALLBACK` is `432`, consisting of `184` MLA prefetch + `184` index prefetch fallbacks for `prefix_len=0`, and `64` `current_reuse` fallbacks for `draft_partial_current_reuse_disabled`.
- Recent tail also shows heavy owner-lane device eviction before large CP load-back when device lanes are nearly exhausted, e.g. load-back of `107520` tokens required evicting `135360` physical tokens to fix per-lane deficits. This is relevant to performance/device-cache churn, but it does not explain decode EAGLE rejection by itself.

Implication:

- The new logs strengthen, rather than weaken, the previous conclusion: accept length dropping toward `1` is a cache-hit-associated EAGLE draft rejection problem after transfer, not missing draft-buffer registration and not solely async-prefetch/tiny-suffix behavior.
- The next root-cause target is the prefill-to-decode EAGLE state contract on cache-hit paths: target/draft hidden-state capture, draft KV content for cached prefix/current suffix, and page-aligned CP local/global position trimming. The remaining suspicious asymmetry visible in logs is that target MLA uses partial-current reuse while draft cache-hit current reuse still explicitly falls back to full materialization.

### C36. 2026-05-30 runtime-validation correction: growing log file is not proof of a new code version

Correction:

- A log file under `/mnt/beegfs/cjy/log/` continued to grow, but the active prefill PID was still `2586666` and the active decode PIDs were still `2446863` / `2208897`. That means the inspected evidence came from the already-running old process, not from a restarted process with any subsequent code changes.
- Therefore C35 must be treated as evidence for the old live runtime only. It is not validation of any newer local code or newer remote-synced code.

Guardrail:

- For future remote validation, do not infer a new runtime from log freshness or file name alone. Before using logs as validation evidence, record all three:
  1. process PID and start/elapsed time,
  2. runtime environment flags relevant to the change,
  3. code identity, preferably git commit or an explicit startup marker from the running checkout/container.
- If the process was not restarted after a code fix, report that explicitly and wait for/user-triggered traffic only after saying the new code is ready to restart.

### C37. 2026-05-30 cache-hit EAGLE low-accept investigation narrowed to draft HiCache/materialize contract

Evidence from the still-running process recorded in C36, not a restarted/new-code validation:

- Decode samples with high cache-hit prefixes continue to show many `accepted_draft=0` entries.  Examples include page-floored cache hits such as `origin=116609 cached=116544`, plus other concurrent high-hit requests (`origin=117775 cached=117760`, `81781/81728`, `79695/79680`, `93640/93632`).
- Aggregate parsing on the same logs still shows the symptom after transfer pressure drains: `#transfer-req:0` windows contain many `accept len <= 1.05`, while no-cache windows are materially healthier.
- Startup diagnostics already proved target/draft KV buffers and target/draft state buffers are registered in the disaggregation KV manager, so the simple “draft transfer surfaces are missing” hypothesis is rejected for this run.
- Prefill tail shows cache-hit target current reuse active with async prefetch (`used_prefetch=True`) and CP page-floored cached prefixes, followed by CP backup final ack and prepared backup logs.  One inspected tail had `extend_lens=[8331]` but backup `logical_len=8330`; this is a candidate off-by-one/contract point to verify before changing behavior.

Current narrowed hypotheses:

1. Draft HiCache load/backup may not be fully symmetric with target under CP shared KV cache hit, even though draft buffers are registered.
2. Draft cache-hit materialization may diverge from target because target uses partial-current reuse/splice while draft currently disables partial-current reuse and falls back to full materialization.
3. Decode-side EAGLE prebuilt initialization may have an output-token off-by-one interaction with disaggregated cache-hit metadata, but this must be compared against the existing chunk-cache/prebuilt contract before modifying it.

Guardrail:

- Do not add another cache/prefetch behavior change from accept-length logs alone.  First prove where draft prefix/current KV or transferred EAGLE state diverges from the target contract.

### C38. 2026-05-30 anti-loop investigation ledger: checked points and do-not-repeat list

Purpose:

- This section is the read-first checkpoint for the current EAGLE/CP-HiCache low-accept investigation.  Before re-scanning broad files, read C29-C38 and only inspect a point again if new runtime evidence contradicts it.

Checked / rejected points:

1. `omx explore` is currently unusable in this checkout/runtime: it fails before doing repository lookup with `Cargo.lock` version `4` unsupported by the installed Cargo.  Use narrow shell searches until OMX is repaired; do not spend more investigation cycles on the same failure.
2. The active evidence in C35-C37 came from an old still-running process unless PID/start-time/code identity proves otherwise.  Fresh log growth alone is not proof of a restarted runtime.
3. The simple hypothesis “draft KV/state transfer surfaces are not registered” is rejected for the EAGLE accept-debug run: startup diagnostics showed target KV, draft KV, target state, and draft state buffers visible in the disaggregation KV manager.
4. The observed `extend_lens=N+1` vs HiCache backup `logical_len=N` in EAGLE paths is currently explained by bigram radix keys: `convert_to_bigram_key([t0..tk])` has length `len(tokens)-1`.  Do not treat this as an off-by-one root cause unless a decode-side contract requires token-count rather than bigram-count metadata.
5. CP HiCache target/draft host reservation is symmetric at the allocator level: `reserve_write_cp()` page-pads once, computes one owner pattern, allocates target host indices and draft host indices for the same physical owned positions, and stores both in `CpHiCacheNodeMetadata`.
6. CP HiCache target/draft backup is symmetric at the layer operation level: `submit_write_cp_layer()` submits target D2H and draft D2H on the same write stream and only queues final ack after all required target layers and all draft layers complete.
7. CP HiCache target/draft load-back is structurally symmetric for owned ranks: `load_cp()` replays the saved page-owner vector through `alloc_pages_with_owners()`, queues target `load_queue` and draft `draft_load_queue` against the same physical device indices, and `start_loading()` loads draft layer 0 before target layer 0 completes.  This does not yet prove decode correctness, but it means “draft load queue is never populated” is not supported by code inspection.
8. Zero-owned CP ranks currently queue a zero-length target load op to keep load ACK rows aligned, but do not queue a draft zero-length op.  Since they own no draft bytes this is likely OK, but if a future hang points at load ACK/node-id symmetry, this is the exact edge to revisit.
9. The remaining strongest code-level asymmetry is not host allocation/load/backup; it is MLA materialization/current-reuse: target cache-hit extend may use partial-current reuse, while EAGLE/NextN draft cache-hit extend intentionally returns false from `should_reuse_current_extend_kv()` with reason `draft_partial_current_reuse_disabled` and then falls through to full materialization.
10. Current non-prefetch partial-current compose is no longer a silent fallback for target: when no async prefetcher exists it does synchronous prefix materialize + current splice; if the prefix is not one positive page-aligned prefix, it fail-fast with `[CP_SHARED_KV_FAIL_FAST][mla_partial_current_sync]`.
11. Async MLA/index prefetch attention-window wait is intentionally deferred to consume-time in the current design; stale tests expecting attention-window wait/launch were updated earlier.  Do not re-add attention-window waits without proving consume-time ordering is wrong.
12. The one-page allocator leak from repeated tiny EAGLE exact hits was already root-caused and fixed by freeing exact duplicate CP HiCache tail pages with `include_partial_tail=True` on duplicate insert.  Do not revisit this unless the memory checker again reports an exact page-size accounting gap under the new code.

Open checks still worth doing:

1. Inspect the full decode traceback around the latest `Scheduler hit an exception` in decode logs before changing cache logic again.  A concrete traceback can supersede the accept-length correlation hypotheses.
2. Trace the EAGLE cache-hit prefill-to-decode state contract: target hidden/topk capture, draft hidden capture, draft KV/state transfer, and decode prebuilt initialization.  The key question is why cached-prefix decode rejects draft tokens even after transfer drains.
3. Decide whether draft cache-hit full materialization is semantically equivalent to target partial-current reuse.  If not, draft needs an explicit same-layer current-reuse contract or a fail-fast guard, not another silent fallback.

### C39. 2026-05-30 remote process stop invalidates connection-closed traceback as root-cause evidence

Correction from operator:

- The remote process observed around the latest decode exceptions was manually stopped by the operator.
- Therefore decode-side `RuntimeError: gloo ... Connection closed by peer` and paired `KVTransferError(...): Aborted by AbortReq` records in the old logs are downstream shutdown/abort symptoms, not a valid root-cause signal for the EAGLE accept-length collapse.

Implication:

- Do not use those shutdown tracebacks to drive cache or EAGLE code changes.
- Continue root-cause work from the stable evidence already recorded: cache-hit decode windows show low EAGLE accept length while draft KV/state transfer surfaces are registered, and the main unproven asymmetry remains cache-hit target partial-current reuse versus draft full materialization / draft current-reuse disablement.

### C40. 2026-05-30 EAGLE/CP-shared-KV prefill contract checked points

Purpose:

- This section records the code facts checked after C39 so the next continuation does not repeat the same broad reads.

Checked points:

1. `_in_seq_collect_last_token()` is not currently an obvious padded-tail bug when NSA in-seq CP split is enabled.  It uses `actual_token_count=sum(forward_batch.extend_seq_lens_cpu)` and `_get_in_seq_last_token_owner_and_offset(split_list, cp_size, actual_token_count)`, so the collected last hidden state is based on the valid extend length, not the padded physical tail.
2. `can_cp_split()` now skips NSA in-seq CP split for CP shared-KV cache-hit suffixes whose padded physical page count is below `cp_size`.  When this happens, `forward_batch.nsa_cp_metadata` is never populated and `nsa_use_prefill_cp()` becomes false.  The model then runs the unsplit/full-token path, while the NSA backend can still use CP shared-KV materialization because that gate is `forward_batch.uses_cp_shared_kv`, not `nsa_use_prefill_cp()`.
3. In the no-NSA-split path, MLA direct local write from `forward_mla._maybe_write_cp_shared_local_mla_kv()` is not used because it is called only under `nsa_use_prefill_cp()`.  The later NSA backend save path still filters logical `out_cache_loc` to owned physical locs through `_maybe_filter_shared_mla_kv_write()` before writing.  This is the expected fallback write route for short suffixes without in-seq CP metadata.
4. `ForwardMode.is_context_parallel_extend()` excludes `DRAFT_EXTEND` v1, but prefill-time EAGLE draft extension does not set the schedule batch to `DRAFT_EXTEND`; it keeps `ForwardMode.EXTEND` with `EagleDraftInput`.  Therefore prefill draft can still create NSA CP metadata and hit the draft partial-current guard.  Decode-time `forward_draft_extend_after_decode()` explicitly sets `ForwardMode.DRAFT_EXTEND`, so that later path is different and should not be conflated with prefill draft extend.
5. The logged `draft_partial_current_reuse_disabled` fallback is therefore specifically meaningful for prefill-time EAGLE draft cache-hit extends (`forward_mode=EXTEND`, `spec_info=EAGLE_DRAFT`), not for ordinary decode `DRAFT_EXTEND` after verification.
6. Target hidden capture for CP draft shared-KV is split-aware: target forward stores `forward_batch.draft_hidden_states` before the final CP collect.  If CP split is active, this is CP-local hidden; if CP split is skipped, it is full-token hidden.  Draft `_get_cp_local_spec_hidden_states()` accepts either shape: it returns local hidden unchanged or CP-splits full hidden when metadata exists.
7. `LogitsProcessor` compact-hidden behavior is compatible with CP narrow output: when the hidden tensor already has one row per request, it returns logits directly and returns hidden states only when `capture_hidden_mode=LAST`.  The target CP shared-KV path bypasses the need for full hidden by using the separate `draft_hidden_states` side channel.

Open after C40:

- The remaining unproven point is still semantic, not a missing allocator/transfer surface: for prefill-time EAGLE cache-hit, target may splice current KV with partial-current reuse while draft currently full-materializes.  Need prove whether those two produce equivalent draft logits/topk/hidden for the transferred prebuilt decode state.  If not equivalent, implement draft same-layer current reuse or fail-fast that path; do not add silent fallback.
- Need a focused unit/instrumented check around prefill-time EAGLE draft cache-hit materialization, not another pass over HiCache target/draft host allocation.

### C41. 2026-05-30 EAGLE disagg/cache contract checked points after manual stop

Purpose:

- Continue the C40 investigation without re-reading the same allocator and transfer paths.  These points were checked after the operator noted the previous process had been manually stopped.

Checked points:

1. The newest prefill log at `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_184808.log` is not accept-length validation evidence.  The active prefill process PID `2639412` started at `2026-05-30 02:48:08 CST`; the inspected tail only shows warmup and `The server is fired up and ready to roll!`, with no new decode traffic evidence in that file.
2. `send_kv_chunk()` transfers target/draft KV pages for `req_to_token[req_pool_idx, start_idx:end_idx]`, where the default final `end_idx` is `min(len(req.fill_ids), len(req.origin_input_ids))`.  NSA state pages use the same `:seq_len` origin/fill span.  This confirms prefill does not intentionally transfer generated output-token KV as part of the initial origin chunk; the first generated token is transferred through metadata (`output_id`, topk, hidden) and decode appends it on commit.
3. Decode pre-allocation mirrors that contract: `_pre_alloc()` allocates `origin_input_len` before metadata commit; `_commit_transfer_to_req()` appends the transferred `output_id` and stores transferred EAGLE `topk_p/topk_index/hidden_states_tensor`.  This makes the prefill/decode origin-token transfer length consistent at the scheduler/metadata level.
4. EAGLE radix using one fewer key than token count is also the existing base radix contract, not a CP-only change.  `RadixCache.cache_unfinished_req()` comments explicitly state that in EAGLE mode “bigram keys will only cache len - 1 kv indices”.  Keep treating `logical_len = token_count - 1` as intended unless a draft-KV value mismatch is proven.
5. CP HiCache `load_cp()` internally allocates padded physical pages and returns only `visible_device_indices` sliced to each node's valid length.  Therefore `HiRadixCache.load_back()` setting `loaded_node.value` using logical `host_len` offsets is correct for the returned tensor; the padded offset handling already happened inside `load_cp()`.
6. The remaining unproven correctness gap is therefore narrower: whether EAGLE/NextN draft prefill cache-hit attention should continue using full materialization from the draft pool while the target path uses partial-current compose, or whether the draft path must get the same current-suffix compose contract.  This is not answered by allocator/load symmetry alone.

Next check:

- Inspect the draft full-materialize path against the partial-current compose path for prefill-time EAGLE cache hits.  If full materialization can read stale/missing current suffix rows before draft attention, switch draft cache-hit prefill to the same explicit current-suffix compose contract or fail fast; do not add another silent fallback.

### C42. 2026-05-30 draft cache-hit current-reuse contract change

Root-cause update:

- The previous draft guard in `should_reuse_current_extend_kv()` made prefill-time EAGLE/NextN draft cache-hit suffixes use full materialization from the persistent draft pool while the target path used prefix materialize + fresh current-suffix splice.
- That is the wrong contract boundary: disabling draft next-layer async prefetch is still correct because the draft model has no real next decoder layer, but disabling same-layer current suffix reuse changes the KV source visible to draft attention.
- The current fix removes the draft-specific current-reuse fallback.  Target and prefill-time draft now share the same decision: current-only or cache-hit partial-current batches should materialize the page-aligned prefix and splice the freshly rebuilt current suffix.

Guardrails:

- This does not re-enable EAGLE/draft async prefetch.  `cp_shared_kv_should_prefetch_next_layer()` still returns `False` for draft input.
- If a future hang reappears in draft MoE/DeepEP, investigate current-suffix rebuild/compose ordering directly; do not silently restore full-materialize fallback because that recreates the target/draft asymmetry.

Verification status:

- Local py_compile passed for the changed runtime/test files.
- Remote container verification on `g0034:/sgl-workspace/sglang-tai` passed:
  - py_compile for `cp_shared_kv_runtime.py` and `test_cp_shared_kv_runtime.py`
  - focused draft-current-reuse and draft-prefetch tests: `2 passed`
  - full `test_cp_shared_kv_runtime.py`: `73 passed, 5 warnings, 2 subtests passed`
  - related CP shared-KV/page-aligned suite: `test_nsa_cp_utils.py`, `test_cp_shared_kv_layout.py`, and `test_cp_shared_kv_runtime.py`: `128 passed, 5 warnings, 2 subtests passed`
- ETE/cache-hit accept-length validation is still not done for this change because the last inspected prefill log only showed warmup and the operator controls traffic generation.

### C43. 2026-05-30 remote-runtime identity after draft current-reuse sync

Checked point:

- The changed runtime/test files were synced to `g0034:/mnt/beegfs/cjy/sglang-dev` and their SHA-256 hashes match the local checkout.
- The active prefill process on `g0034` was PID `2639412`, started at `2026-05-30 02:48:08 CST`, which predates this sync.  Therefore that process cannot validate the draft current-reuse fix until it is restarted.
- The latest visible decode accept-debug log snippets are still evidence from the old runtime lineage.  They remain useful for root-cause correlation, but not for validating C42.

Guardrail:

- Do not interpret any cache-hit accept-length numbers from the currently running old process as success or failure of C42.  Require process start time after the sync, or an explicit code identity marker, before using ETE logs for this change.

### C44. 2026-05-30 EAGLE prebuilt top-k metadata shape checked point

Checked point:

- The current launch uses `--speculative-algorithm EAGLE` without `--enable-multi-layer-eagle`, so the worker is `EAGLEWorkerV2`, not `MultiLayerEagleWorkerV2`.
- In this mode, prefill draft `_draft_extend_for_prefill()` only produces one draft top-k state for the next decode draft call.  `decode_schedule_batch_mixin.process_prebuilt()` therefore intentionally slices `req.output_topk_p[:speculative_eagle_topk]` and does not multiply by `speculative_num_steps`.
- The decode debug log prints the full metadata buffer shape (for example `(16,)`) because the disaggregation metadata buffer is over-provisioned; process-prebuilt uses only the configured top-k prefix.  This is not by itself evidence of a metadata shape bug.

Rejected:

- Do not change prebuilt `num_states` to `topk * speculative_num_steps` for the non-multi-layer EAGLE path unless the launch also enables `enable_multi_layer_eagle`.  That would feed extra uninitialized/irrelevant metadata into `EAGLEWorkerV2.draft()`.

### C45. 2026-05-30 new runtime did not recover; root-cause moved to decode metadata-buffer lifetime

Runtime evidence:

- New prefill process PID `2655723` started at `2026-05-30 03:18:38 CST`, after the C42 code sync.
- Latest logs:
  - prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_191838.log`
  - decode0: `/mnt/beegfs/cjy/log/decode0_20260529_191842.log`
  - decode1: `/mnt/beegfs/cjy/log/decode1_20260529_191845.log`
- The C42 change is active enough to remove the old fallback: `draft_partial_current_reuse_disabled=0` in the new prefill log.
- Accept length still did not recover: parsed decode batches show average accept length about `1.35`, median about `1.25`, and many windows at `<=1.05`.  Transfer-free windows remain low as well.

New root cause:

- `DecodeTransferQueue._commit_transfer_to_req()` assigned `req.output_topk_p`, `req.output_topk_index`, and `req.hidden_states_tensor` directly from `metadata_buffers.get_buf(idx)`.
- `metadata_buffers.get_buf(idx)` returns views into reusable metadata-buffer rows.
- `DecodeTransferQueue.pop_transferred()` frees `metadata_buffer_index` immediately after `_commit_transfer_to_req()` returns, but the transferred request can sit in the scheduler waiting queue before `process_prebuilt()` consumes the EAGLE top-k/hidden state.
- Therefore later transfers can overwrite the same metadata slot before decode prebuilt consumes it.  The scalar output id and cached-token count are copied with `.item()`, but the EAGLE draft state was not owned.  This explains why transfer and registration look correct while the first decode draft state is often wrong under concurrent/high-cache-hit traffic.

Fix:

- Clone the EAGLE metadata tensors at commit time before freeing the metadata slot:
  - `output_topk_p.clone()`
  - `output_topk_index.clone()`
  - `output_hidden_states.clone()`
- Added a unit test that mutates the original metadata-buffer tensors after commit and verifies the request still owns the original values and different storage.
- Remote container verification on `g0034:/sgl-workspace/sglang-tai` passed:
  - py_compile for `decode.py` and `test_decode_queue_compaction.py`
  - focused clone-lifetime test: `1 passed`
  - full `test_decode_queue_compaction.py`: `10 passed, 5 warnings`

Guardrail:

- Do not replace the clone with a view unless the allocator lifetime is changed so the metadata slot is held until after `process_prebuilt()` consumes the request.

### C46. 2026-05-30 decode EAGLE metadata lifetime must be ownership-based, not clone-based

Correction:

- The C45 clone fix is only a safety workaround.  It protects the EAGLE top-k/hidden tensors from metadata-slot reuse, but it adds an extra device/host copy on the transfer hot path and hides the real ownership bug.
- The better contract is: when decode disaggregation commits an EAGLE transfer, the request owns the metadata-buffer slot until decode prebuilt has consumed `req.output_topk_p`, `req.output_topk_index`, and `req.hidden_states_tensor`.
- Therefore `DecodeTransferQueue.pop_transferred()` must not free a successful EAGLE request's `metadata_buffer_index` immediately.  It should hand the slot index to the `Req`; `process_batch_result_prebuilt()` should free it after `process_prebuilt()` has already built `EagleDraftInput`.

Rejected:

- Do not keep clone as the final fix.  It is correct but less efficient and does not encode the real lifetime contract.

Risk:

- Holding slots can reduce metadata-buffer availability when many transferred requests wait in the scheduler queue.  That is expected backpressure because those requests still reference live metadata-buffer rows.
- Non-spec decode does not need the EAGLE tensor views, so it can continue freeing metadata slots at transfer commit.

### C47. 2026-05-30 held-slot implementation and verification status

Implemented:

- `_commit_transfer_to_req()` now stores EAGLE metadata as views and attaches `metadata_buffer_index` to the transferred `Req`.
- `pop_transferred()` keeps successful EAGLE metadata slots allocated; non-spec success and failed/corrupt transfers still free immediately.
- `process_batch_result_prebuilt()` frees any held decode metadata slot after prebuilt has consumed the EAGLE metadata.
- `Scheduler.abort_request()` also frees held decode metadata for requests aborted while waiting before prebuilt.

Verification:

- Remote red tests before implementation failed as expected:
  - successful EAGLE transfer incorrectly freed metadata slot immediately
  - prebuilt output processing did not free held metadata slot
- Remote container verification after implementation passed:
  - py_compile for `decode.py`, `scheduler_output_processor_mixin.py`, `scheduler.py`, and `test_decode_queue_compaction.py`
  - focused held-slot tests: `2 passed`
  - full `test_decode_queue_compaction.py`: `11 passed, 5 warnings`

Open:

- ETE accept-length recovery still requires a restarted runtime with this code.  Do not use logs from a process started before this sync to validate this fix.

### C48. 2026-05-30 decode EAGLE metadata slot release narrowed to consume point

Correction:

- C47 released held decode EAGLE metadata slots in `process_batch_result_prebuilt()`, after the prebuilt batch had already run forward.  That is correct but holds metadata slots longer than necessary.
- The real consume point is `get_new_prebuilt_batch()` immediately after `new_batch.process_prebuilt(...)` returns.  At that point `process_prebuilt()` has already materialized `EagleDraftInput` from `req.output_topk_p`, `req.output_topk_index`, and `req.hidden_states_tensor`; the reusable disaggregation metadata slot is no longer needed.

Implemented:

- `get_new_prebuilt_batch()` now frees held decode metadata slots in a `finally` block immediately after `process_prebuilt()`.
- `process_batch_result_prebuilt()` no longer owns metadata-slot cleanup.
- The metadata-free helper remains shared for the consume-point release and waiting-queue abort cleanup.

Verification:

- Remote red test first failed because `get_new_prebuilt_batch()` did not free metadata slots after `process_prebuilt()`.
- Remote container verification after implementation passed:
  - py_compile for `decode.py`, `scheduler_output_processor_mixin.py`, `scheduler.py`, and `test_decode_queue_compaction.py`
  - focused lifecycle tests: `3 passed`
  - full `test_decode_queue_compaction.py`: `11 passed, 5 warnings`

Open:

- ETE/runtime validation still needs a fresh process started from this synced code.

### C49. 2026-05-30 remote log observation after C48 sync

Runtime identity:

- Active prefill process on `g0034` was PID `2729613`, started at `2026-05-30 04:36:56 CST`.
- Latest C48 source sync happened after that process started: remote source mtimes around `2026-05-30 04:43:56-04:43:58 CST`.
- Therefore this process cannot validate the C48 consume-point metadata-slot release.  It may only validate the earlier code state present at launch.

Latest logs inspected:

- Prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_203656.log`
- Decode0: `/mnt/beegfs/cjy/log/decode0_20260529_203700.log`
- Decode1: `/mnt/beegfs/cjy/log/decode1_20260529_203703.log`

Observed:

- Process is still alive and logs are still moving; no fatal traceback, health-check failure, memory leak, or OOM was found in the latest scan.
- Errors are mostly transfer aborts (`KVTransferError(...): Aborted by AbortReq`) and startup/import warnings.
- Decode accept remains low in this runtime:
  - decode0 batch accept len avg `1.511`, p50 `1.36`, p10 `1.02`, p90 `2.19`
  - decode1 batch accept len avg `1.457`, p50 `1.30`, p10 `1.02`, p90 `2.16`
  - EAGLE debug avg_accept is much lower because it counts accepted draft tokens only: decode0 avg `0.415`, decode1 avg `0.298`; many lines have zero accepted drafts.
- EAGLE metadata presence itself is not missing: sampled debug lines show `has_pd_hidden=True`, `pd_hidden_shape=(6144,)`, `pd_topk_shape=(16,)`.
- Prefill is actively doing owner-lane eviction and CP shared-KV current reuse:
  - `owner-lane evict END`: 768 lines
  - `forward_partial_current_reuse`: 2688 lines
  - `used_prefetch=True`: 1072 lines
  - `used_prefetch=False`: 1616 lines
  - `create_skip reason=prefix_below_min`: 1472 lines

Interpretation:

- Do not use this process to judge C48.
- If the earlier held-slot fix was active in this process, low accept length is not fully explained by missing EAGLE metadata lifetime alone; next investigation should compare a fresh C48 runtime and then inspect EAGLE state correctness/content, not just tensor presence.

### C50. 2026-05-30 output_len=0 is confirmed by router/client; server-side checked points

New user-provided fact:

- `output_len=0` has been confirmed from router/client logs.  Treat it as a
  real external symptom even if the currently inspected SGLang server logs do
  not print full response bodies.

Server-side evidence checked so far:

- The latest valid decode logs around the previous failure window contain a
  burst of `Decode transfer failed ... Aborted by AbortReq` at
  `2026-05-29 21:01:40`, and the paired prefill log contains matching
  `Prefill bootstrap failed` / `Prefill transfer failed` aborts at
  `2026-05-29 21:01:41`.
- A request aborted before any decoded token can produce a zero-length final
  output at the scheduler boundary (`stream_output_generation()` reports
  `completion_tokens=len(req.output_ids_through_stop)`).  The OpenAI serving
  layer should normally convert scheduler abort finish reasons to HTTP errors,
  so a client-observed successful empty response still needs a request-id/rid
  correlation or a server-side zero-output warning to determine whether the
  empty result is `abort`, `length=0`, stop-trimming, or router/client handling.
- The newer prefill process PID `2763878` started at `2026-05-30 05:01:46 CST`
  from the synced code, but at the time of this check it was still in startup /
  weight-load/warmup logs (`sglang_cp_hicache_20260529_210146.log`).  It cannot
  yet validate whether C48/C49 fixed or changed the output-len symptom.

Do-not-repeat:

- Do not conclude “server did not output zero” only because the SGLang log does
  not contain `completion_tokens=0`; those bodies are usually not logged.
- Do not treat router `Prefill server failed (CRITICAL)` as an independent
  router bug when it aligns with prefill/decode KV-transfer aborts.

Next useful evidence:

- Add a low-frequency/anomaly-only server warning at the scheduler output
  boundary for finished generation responses with `completion_tokens == 0`,
  including `rid`, `finish_reason`, `max_new_tokens`, `origin_len`, cached
  tokens, disaggregation mode, and bootstrap room.  This should fire only on the
  confirmed anomalous path and should be removed or gated after root cause is
  found.

### C51. 2026-05-30 split hypothesis for output_len=0

New user-provided hypothesis:

- `output_len=0` may be caused by a split path.

Checked points:

- There are two distinct split mechanisms in the current failure surface:
  1. NSA in-seq CP split (`can_cp_split()` / `build_page_aligned_in_seq_split_list()`),
     which affects forward compute distribution.
  2. Radix/HiCache tree split (`HiRadixCache._split_node()`), which affects
     device/host cache residency and prefix matching.
- C24 already disables NSA in-seq CP split for cache-hit suffixes with fewer
  physical suffix pages than CP lanes.  This removes the known `extend_len=65`
  mostly-zero-lane split path, but it does not address long suffixes or radix
  structural splits.
- Current radix exact-valid-tail extension handling floors the matched prefix
  to the previous page boundary.  During insertion it then splits the old
  valid-tail node and can leave two children under the same page-boundary parent
  whose keys start at the same logical tail page, e.g. with `page_size=4`:
  old child `(4, 5)` and new longer child `(4, 5, 6, 7)`.
- This overlapping-child shape is not yet proven to produce `output_len=0`, but
  it is a real radix invariant risk: two children can represent overlapping
  logical token ranges with independent device/host/draft HiCache state.  Under
  eviction/load-back or EAGLE target/draft mirror pressure, that can create
  target/draft cache-state divergence or unnecessary transfer/abort pressure.
- The latest inspected runtime did not include the new `OUTPUT_ZERO_DEBUG`
  scheduler boundary warning, so server-side classification of confirmed
  client/router `output_len=0` is still pending a fresh process.  Existing logs
  still show the direct zero-output-compatible server symptom as large bursts of
  prefill/decode `KVTransferError(...): Aborted by AbortReq` followed by decode
  losing the prefill connection.

Next checks:

- Add a focused radix regression that demonstrates the overlapping-child shape
  for exact valid-tail extension and then replace it with a page-granular policy
  that sacrifices the old sub-page tail instead of keeping two overlapping
  children.
- Keep the scheduler `OUTPUT_ZERO_DEBUG` anomaly warning until a fresh runtime
  confirms whether client/router `output_len=0` corresponds to abort,
  max_new_tokens=0/length finish, stop trimming, or another finish reason.

### C52. 2026-05-30 exact valid-tail split currently preserves stale sub-page child

Finding:

- The existing focused unit test `test_cp_insert_extends_from_page_boundary_after_exact_valid_tail`
  encodes the risky behavior from C51 as expected behavior: after an old valid-tail
  CP node `(0..5)` is extended by a new request `(0..9)`, the radix tree keeps both
  the stale old child `(4,5)` and the new longer child `(4,5,6,7,8,9)` under the
  page-boundary parent `(0..3)`.
- This violates the page-granular cache contract we now want for CP HiCache: a
  logical sub-page tail is not an independent cache-management unit once a request
  extends beyond it.  The old tail page should be sacrificed/freed, and the new
  suffix should own that page from the boundary.

Next implementation guardrail:

- Add/adjust the regression first so the expected tree has only the new suffix
  child after exact-valid-tail extension.  Then implement a narrow CP-only prune
  path for the stale tail child, with pending/in-flight backup split still fail-fast
  or deferred instead of silently mutating an in-flight node.

### C53. 2026-05-30 stale-tail pruning must cover backed partial-tail splits too

Finding:

- The exact-valid-tail extension case is only one instance of the same bug.
  `test_cp_insert_floors_backed_tail_split_to_page_boundary` shows a backed CP
  HiCache node `(0..5)` split by a shorter/new partial request `(0..4)`.  The old
  test also preserved the stale child `(4,5)` beside the new child `(4)`, creating
  the same overlapping-page state.
- Therefore the prune condition cannot be limited to exact valid-tail extension.
  Whenever CP split length is floored from inside a valid/padded tail page to the
  previous page boundary, the stale tail subtree created by `_split_node()` must
  be pruned unless it is protected or has an in-flight backup.

Guardrail:

- Probe-only paths may continue to floor without mutating.  Mutating match/insert
  paths must prune stale tails after page-floor split; protected/in-flight tails
  must defer/fail rather than silently keeping overlapping children.

### C54. 2026-05-30 stale CP tail pruning implemented and tested

Implemented:

- CP mutating radix match/insert paths now detect when a split length was floored
  from inside a page to the previous page boundary.
- After `_split_node()` creates the page-boundary parent, the stale sub-page tail
  subtree is pruned instead of being kept as an overlapping child.  This applies
  both to exact valid-tail extension and backed partial-tail split cases.
- The prune path first checks the stale subtree for lock refs, host refs, pending
  backup, or ongoing write-through state.  Such in-flight/protected tails defer
  or fail via the existing pending-split path rather than silently mutating cache
  state used by active work.
- Pruning frees device indices and CP host/draft metadata when the corresponding
  cache-controller eviction hooks are available, then unlinks the stale subtree
  and refreshes leaf state when the cache was fully initialized.

Verification:

- Remote py_compile passed for `python/sglang/srt/mem_cache/hiradix_cache.py` and
  `python/sglang/srt/managers/scheduler_output_processor_mixin.py` in container
  `/sgl-workspace/sglang-tai`.
- Remote targeted red tests first failed on the old overlapping-child behavior.
- Remote full focused suite passed after the fix:
  `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py`
  -> `97 passed, 5 warnings`.

Remaining risk:

- This removes one concrete split-induced overlapping-cache invariant violation,
  but it does not yet prove the external `output_len=0` symptom is fixed.  A fresh
  ETE run still needs to check for `[OUTPUT_ZERO_DEBUG]`, decode accept length,
  and router/client zero-output correlation.

### C55. 2026-05-30 latest split-prune runtime: zero-output not seen in server logs, EAGLE accept still collapsed

Runtime identity:

- New prefill process on `g0034` PID `2796105` started at `2026-05-30 05:36:48 CST`, after the split-prune and `OUTPUT_ZERO_DEBUG` source sync (`05:33-05:34 CST`).
- Decode processes on `g0035`/`g0036` started at `05:36:52/05:36:55 CST` and see the same synced source mtimes.
- Latest logs inspected:
  - prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_213648.log`
  - decode0: `/mnt/beegfs/cjy/log/decode0_20260529_213652.log`
  - decode1: `/mnt/beegfs/cjy/log/decode1_20260529_213656.log`

Checked result:

- No `[OUTPUT_ZERO_DEBUG]` line was found in prefill/decode logs, and no server-side `completion_tokens: 0` / `output_len=0` text was found in these logs.  This does not prove the router/client symptom is gone, but the scheduler boundary did not observe a finished generation with zero visible tokens in the inspected log window.
- No current prefill/decode `KVTransferError`, transfer abort burst, traceback, health-check failure, or scheduler exception was found in this log window.  The only `out of memory` text was warmup max_m reduction, not runtime OOM.
- EAGLE accept remains bad in the inspected decode logs:
  - decode0: `17649` accept-debug rows, average `avg_accept=0.073`, median `0.0`, `15650` zero rows, `17544` rows at `<=1.05`; last 200 rows average `0.135`.
  - decode1: `17541` accept-debug rows, average `avg_accept=0.065`, median `0.0`, `15569` zero rows, `17495` rows at `<=1.05`; last 200 rows average `0.030`.
- Metadata presence is not missing in the samples: `has_pd_hidden=True`, `pd_hidden_shape=(6144,)`, `pd_topk_shape=(16,)`, no `has_pd_hidden=False` / `pd_topk_shape=None` lines.  `spec_topk_shape=(0,1)` appears occasionally but is not the dominant count; most rows have `(1,1)` or `(2,1)`.

Interpretation:

- Split-prune removes a real overlapping-radix invariant violation, but it did not recover EAGLE accept length in this run.
- The current accept collapse is more likely in the semantic correctness/content of EAGLE prebuilt state or draft/target KV alignment than in missing metadata lifetime or gross transfer failure.  Next debugging should compare the actual prebuilt top-k token/content and draft input state against a non-HiCache or non-CP baseline, not just tensor shape/presence.

### C56. 2026-05-30 accept-length collapse investigation: current checked facts

Checked facts:

- Current decode launch uses non-v2 `EAGLEWorker`, not `EAGLEWorkerV2`: `SpeculativeAlgorithm.create_worker()` selects V2 only when overlap scheduling is enabled, while the inspected decode server args have overlap disabled.  V2-only hypotheses around `prepare_for_v2_draft()` are therefore not applicable to the latest run.
- Latest aggregate `Decode batch` accept length remains low but is not literally always 1: in the latest decode logs the average accept length is about `1.33`, median about `1.23-1.25`, with many windows at `<=1.05`.  The per-request `EAGLE_ACCEPT_DEBUG` rows are intentionally zero-biased because zero-accept cases are logged much more often than non-zero samples, so they are useful for anomaly examples but not for true distribution estimates.
- Decode processes are not running HiCache/NSA-prefill-CP-shared-KV.  The low accept issue is therefore more likely caused by prefill-transferred target/draft KV or prebuilt EAGLE state content than by decode-side HiCache behavior.
- Metadata lifetime is not the obvious failure: latest samples show `has_pd_hidden=True`, `pd_hidden_shape=(6144,)`, and `pd_topk_shape=(16,)`; missing hidden/top-k metadata is not the dominant failure mode.
- `process_prebuilt()` builds EAGLE prebuilt state from the transferred first output token, top-k, and hidden state, then calls `cache_unfinished_req(req)`.  `fill_ids` is still the prompt/input tokens prepared before transfer; for EAGLE this is plausibly intentional because unfinished bigram cache should not cache the last row that depends on the generated output.  This is a checked contract point, not a fix target unless content evidence proves otherwise.

Next focused check:

- Verify whether partial cache-hit prefill has symmetric current-suffix handling for MLA KV and NSA index KV.  MLA current-suffix reuse was intentionally enabled after partial cache hits; index KV current-suffix reuse may still be restricted to current-only batches.  If index materialization/top-k reads a stale or incomplete persistent index buffer for a cache-hit suffix, EAGLE top-k/hidden can be semantically bad even though all transferred tensors have correct shapes.

### C57. 2026-05-30 accept-length collapse: target logs do not prove draft KV correctness

Checked facts:

- The current `SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH` samples are layer-probe based
  and the active probe evidence is from target layer 2.  EAGLE/NextN draft has a
  single draft layer (`layer_id=0`), so these logs do not prove that draft
  partial-current compose is correct or even which draft path is active.
- Latest prefill log does show many target cache-hit partial-current cases.  For
  small cache-hit prefixes such as `prefix_lens=[320]` (5 pages), the async
  prefetcher is absent because the threshold is 8 pages, and target falls back
  to the synchronous partial-current compose path.  This is expected under the
  current threshold policy and should not be re-read as “prefetch is missing”.
- NSA index current reuse is still current-only: `_can_reuse_current_index_kv()`
  requires `is_current_only_extend_batch()`, so partial cache-hit requests do not
  take the index-current-reuse fast path.  However the indexer stores the current
  suffix local index KV into the persistent pool before materializing the shared
  index buffer for top-k, so the absence of an index partial-current fast path is
  currently a performance risk more than a proven accept-length correctness root
  cause.
- The remaining high-probability correctness surface is draft KV semantic/content
  alignment across prefill cache-hit paths and decode prebuilt consumption:
  decode itself is not running CP HiCache, so wrong draft KV transferred from
  prefill can suppress EAGLE accept while metadata tensor shapes still look
  healthy.

Do-not-repeat:

- Do not use target layer-2 MLA prefetch logs as evidence for EAGLE draft layer
  correctness.  Draft needs its own low-frequency probe or a direct KV/content
  comparison.
- Do not treat missing async prefetcher on a 5-page prefix as a bug; it follows
  the configured 8-page threshold.  The correctness requirement is that sync
  partial-current/full-current still works when prefetch is disabled.

### C58. 2026-05-30 accept-length collapse: prebuilt/draft evidence gaps to close next

Checked facts:

- Disaggregated decode constructs the initial EAGLE running batch through
  `get_new_prebuilt_batch()`: it first calls `req.init_next_round_input(tree_cache)`,
  then `prepare_for_prebuilt()`, then `process_prebuilt()`.
- Because decode has its own radix tree, repeated prompts can enter prebuilt with
  a non-zero decode-side `prefix_indices` even though KV has already been
  preallocated/transferred for the full prompt.  Current logs do not show this
  `pre_len`, so we cannot yet correlate low accept length with decode-side
  prefix reuse.
- `prepare_for_prebuilt()` builds `input_ids` from `fill_ids[pre_len:]`, but the
  transferred `req_to_token` rows are full-prompt rows.  If any later prebuilt or
  EAGLE path consumes `out_cache_loc`, the correct transferred suffix rows start
  at `pre_len`, not at zero.  This is a concrete invariant risk, but current
  non-overlap EAGLE mostly consumes the transferred req-to-token mapping and
  prebuilt EAGLE state after `prepare_for_decode()`, so it still needs runtime
  evidence before being treated as the accept-collapse root.
- The latest target-side CP shared-KV logs still do not identify the draft layer
  path.  We need a draft-layer (`is_draft_input`, layer 0) path log showing
  whether EAGLE draft is using current-only, partial-current sync/prefetch, or
  full materialize on cache-hit requests.

Next instrumentation:

- Under `SGLANG_EAGLE_ACCEPT_DEBUG`, add low-frequency decode prebuilt logs with
  `pre_len`, `extend_input_len`, `fill_len`, `origin_len`, `output_len`, and
  `cached_tokens`.
- Under `SGLANG_EAGLE_ACCEPT_DEBUG`, add low-frequency draft MLA path logs at
  draft layer 0 with branch name, prefix/extend lengths, prefetcher presence,
  current rows, KV rows, and page-table shape.  Do not use the target layer-2
  MLA prefetch probe as draft evidence.

### C59. 2026-05-30 latest EAGLE accept debug run: prebuilt prefix reuse is not active, draft prefill uses partial-current sync

Checked runtime:

- Latest prefill log: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_223451.log`.
- Latest decode logs: `/mnt/beegfs/cjy/log/decode0_20260529_223458.log`, `/mnt/beegfs/cjy/log/decode1_20260529_223500.log`.
- The new `[EAGLE_ACCEPT_DEBUG][prebuilt_prepare]` and `[EAGLE_ACCEPT_DEBUG][draft_mla_path]` probes are present in the latest run.

Findings:

- Decode prebuilt prefix reuse is not active in the inspected window: decode0 had `128` prebuilt debug rows and decode1 had `128`; all had `pre_len=0` even when transferred/cached tokens were large (`cached_max` around `106k-107k`).  Therefore the suspected decode-side suffix offset from non-zero `pre_len` is not the current primary explanation.
- EAGLE accept remains low: latest decode logs are around `avg=1.35-1.38`, `median=1.26-1.29`, `p90=1.81-1.90`, with many windows at `<=1.05`.
- No server-side zero-output marker was found in the latest prefill/decode logs.
- Prefill draft MLA path is now directly observed.  In the latest prefill log there are `128` draft-layer path rows: `48 current_only` and `80 partial_current_sync`.  No draft prefetch path is used, as intended for draft.
- Partial-current draft examples include `prefix_lens=[320]`, large `extend_lens`, `can_current_reuse=True`, `has_prefetcher=False`, `branch=partial_current_sync`.  This proves that the re-enabled draft current-suffix compose path is on the hot path for cache-hit EAGLE prefill.

Next focused question:

- Determine whether EAGLE draft partial-current compose uses the correct valid row/loc contract after `EagleDraftInput.prepare_for_extend()` shifts input ids left and appends the verified output token.  If draft current rows are mapped to target-style `out_cache_loc` rows with an off-by-one semantic shift, the transferred draft KV can have healthy shapes but poor accept length.

### C60. 2026-05-30 likely accept-collapse root: CP-local EAGLE hidden was padded then re-split as if full hidden

Checked code path:

- `EAGLEWorker.forward_target_extend()` stores target hidden for draft in `logits_output.draft_hidden_states` when `SGLANG_CP_DRAFT_SHARED_KV=1`.
- `DeepseekV3ForCausalLM.forward()` captures this side-channel before CP output collection, so the tensor is already CP-local.
- `EAGLEWorker.forward_draft_extend()` passes that CP-local tensor into `EagleDraftInput.hidden_states` for the NextN draft model.
- Before `DeepseekModelNextN.forward()` consumes it, `ForwardBatch.prepare_mlp_sync_batch()` pads `spec_info.hidden_states` to the global padded token count and saves the original tensor in `forward_batch.hidden_states_backup`.
- `DeepseekModelNextN._get_cp_local_spec_hidden_states()` previously checked `spec_hidden_states.shape[0] == full_num_tokens` and then ran `cp_split_and_rebuild_data(...)`.  After the padding above, a CP-local tensor could look like a full-token tensor and get CP-split a second time.

Why this matches the symptom:

- It affects cache-miss and cache-hit EAGLE prefill, not only HiCache partial-current paths, matching the low accept length across runs.
- It preserves tensor shapes and transfer metadata, matching the earlier observation that hidden/top-k shapes were healthy while accept length remained bad.
- It is introduced by the CP-local draft-hidden optimization, not by decode prebuilt prefix reuse; latest logs show `pre_len=0` for all sampled decode prebuilt rows.

Current local fix:

- `DeepseekModelNextN._get_cp_local_spec_hidden_states()` now prefers `forward_batch.hidden_states_backup` when CP draft shared KV is enabled and MLP sync padding changed the visible hidden shape.
- The helper returns CP-local hidden directly, pads only a small local-padding gap, and only splits when the tensor is genuinely full-token hidden.
- A low-frequency `[EAGLE_ACCEPT_DEBUG][cp_draft_hidden]` probe records which branch was taken.

Validation status:

- Local `py_compile` for `python/sglang/srt/models/deepseek_nextn.py` passed.
- Needs remote sync/restart and ETE accept-length verification; no commit yet.

### C61. 2026-05-30 restart after CP-local draft-hidden fix: code path is active but accept length is still low

Checked runtime:

- Latest prefill log: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_122138.log`.
- Latest decode logs: `/mnt/beegfs/cjy/log/decode0_20260530_122148.log`, `/mnt/beegfs/cjy/log/decode1_20260530_122154.log`.
- The `[EAGLE_ACCEPT_DEBUG][cp_draft_hidden]` probe is present in prefill, proving the restarted process loaded the CP-local hidden backup fix.

Findings:

- The CP-local hidden backup path is active.  Example rows show `padded_shape=(10120, 6144)` and `backup_shape=(1280, 6144)`, followed by `using CP-local hidden directly`.
- EAGLE accept did **not** recover in this run:
  - `decode0_20260530_122148.log`: `n=776`, `avg=1.357`, `median=1.2`, `last100=1.323`, `min=1.0`, `max=3.35`.
  - `decode1_20260530_122154.log`: `n=754`, `avg=1.380`, `median=1.2`, `last100=1.418`, `min=1.0`, `max=3.05`.
- Decode prebuilt prefix reuse remains inactive in the sampled window: `128` prebuilt rows per decode log and all sampled `pre_len=0`, even with non-zero `cached_tokens`.
- No server-side zero-output marker or health/scheduler error was found in the latest prefill/decode logs.
- Draft path remains hot and conservative: prefill has `40 current_only` and `96 partial_current_sync` draft-layer debug rows, with no draft prefetcher.
- Only startup/empty-prefix fallback logs were observed (`prefix_len=0` for MLA/index prefetch).  These are not evidence of the current accept collapse.

Conclusion:

- C60 fixed a real CP-local hidden contract bug, but it is not sufficient to restore accept length.  The next root-cause surface is value/content correctness across the EAGLE prefill-to-decode boundary, not tensor shape or decode `pre_len` offset.

Next focused checks:

- Compare the EAGLE draft artifacts produced by prefill and consumed by decode at the request boundary: transferred draft KV content/row ordering, `output_topk_p`, `output_topk_index`, and draft hidden alignment.
- Keep the current shape/branch probes, but add only low-frequency value checksums if needed; avoid broad trace spam.

### C62. 2026-05-30 accept-length collapse: active worker and metadata registration facts

Checked facts:

- The latest server args show `enable_multi_layer_eagle=False` and `disable_overlap_schedule=True`, so the active speculative path is the regular non-v2 `EAGLEWorker`.  Multi-layer EAGLE top-k packing hypotheses are not active for this run.
- Prefill and decode manager startup logs both register `target_kv_bufs=78`, `draft_kv_bufs=1`, `total_kv_bufs=79`, `target_state_type=nsa`, and one appended draft NSA state buffer.  This proves the draft KV/state buffer registration surface is present, but it does not prove values or row ordering are correct.
- Latest decode `prebuilt_prepare` logs remain `pre_len=0`; decode-side radix prefix reuse is not currently shifting the prebuilt input/off-cache mapping.
- The latest accept distribution is still low (`decode0 avg≈1.38`, `decode1 avg≈1.39`) while there are no health/scheduler errors or zero-output markers in the latest prefill/decode logs.

Conclusion:

- The next evidence gap is not worker selection, metadata shape, or missing buffer registration.  We need value-level evidence at the EAGLE prefill-to-decode handoff: top-k token/probability, hidden checksum, and later draft KV content if metadata values match.

Next instrumentation:

- Add low-frequency `[EAGLE_ACCEPT_DEBUG][metadata_set]`, `[metadata_get]`, and `[prebuilt_state]` summaries under `SGLANG_EAGLE_ACCEPT_DEBUG`.
- Keep summaries bounded: request id/room, metadata slot, output id, cached tokens, top-k head, and hidden checksum/head only.  Do not add per-layer or per-token spam.

### C63. 2026-05-30 EAGLE handoff checksum instrumentation added

Local change:

- Added a bounded tensor digest helper in `sglang.srt.disaggregation.utils`.
- `MetadataBuffers.set_buf()` now logs `[EAGLE_ACCEPT_DEBUG][metadata_set]` for the prefill-side EAGLE handoff slot.
- Decode transfer commit now logs `[EAGLE_ACCEPT_DEBUG][metadata_get]` for the decode-side view of the same slot.
- `process_prebuilt()` now logs `[EAGLE_ACCEPT_DEBUG][prebuilt_state]` after stacking top-k and hidden into the initial decode EAGLE state.

Purpose:

- If `metadata_set`, `metadata_get`, and `prebuilt_state` match for the same request/room, the low accept length is unlikely to be caused by metadata corruption and the next target should be draft KV content/row ordering.
- If they diverge, the root is in metadata transfer/lifetime, not draft KV.

Verification:

- Local `py_compile` passed for `disaggregation/utils.py`, `disaggregation/decode.py`, `disaggregation/decode_schedule_batch_mixin.py`, `layers/attention/nsa_backend.py`, and `models/deepseek_nextn.py`.

Not yet verified:

- Needs remote sync/restart and one traffic run with `SGLANG_EAGLE_ACCEPT_DEBUG=1` to compare the actual handoff values.

### C64. 2026-05-30 decode metadata lifetime audit: production free exists, stale aliases remain

Checked facts:

- `SchedulerDisaggregationDecodeMixin.get_new_prebuilt_batch()` calls
  `new_batch.process_prebuilt(...)` and then releases each request's held decode
  metadata slot in a `finally` block via `_free_decode_metadata_index_if_held`.
- The earlier suspicion that the decode metadata slot was never freed in
  production is false; the production release path exists and is covered by
  `test_get_new_prebuilt_batch_slices_waiting_queue_prefix`.
- The release helper currently only frees the allocator index and sets
  `metadata_buffer_index=-1`.  It does not clear `Req.output_topk_p`,
  `Req.output_topk_index`, or `Req.hidden_states_tensor`, so a running request
  can still retain stale views into a reusable metadata row after the slot has
  been returned to the allocator.

Risk:

- The known prebuilt consumer copies top-k and hidden state into GPU tensors
  before the slot is released, so this alias is not yet proven to be the
  accept-length root cause.
- Keeping stale reusable-slot views on `Req` is still an unsafe lifecycle
  contract: later debug, retraction, or future prebuilt paths can observe a
  different request's EAGLE state after slot reuse.

Planned correction:

- Keep the no-clone transfer hot path, but make metadata release also detach the
  request from reusable-slot views by clearing the three EAGLE handoff fields
  after `process_prebuilt` has consumed them.
- Preserve the existing release timing; do not move release back to
  `pop_transferred` and do not reintroduce transfer-time clones unless the new
  handoff checksums prove process-prebuilt copy is not sufficient.

### C65. 2026-05-30 latest remote run still lacks new metadata checksums

Checked facts:

- Remote files under `/mnt/beegfs/cjy/sglang-dev` match the local hashes for the
  new `metadata_set`, `metadata_get`, and `prebuilt_state` instrumentation.
- The currently running logs (`sglang_cp_hicache_20260530_122138.log`,
  `decode0_20260530_122148.log`, `decode1_20260530_122154.log`) contain the
  older `draft_mla_path`, `cp_draft_hidden`, and `prebuilt_prepare` markers but
  no `metadata_set`, `metadata_get`, or `prebuilt_state` markers.  Therefore the
  live process predates the latest handoff-checksum instrumentation.
- Current accept remains low in that old process: decode0 average around 1.38
  and decode1 average around 1.39 accepted length.

Next evidence gate after restart:

- Compare `metadata_set -> metadata_get -> prebuilt_state` for the same request.
  If these match while accept remains low, the next root surface is draft KV
  content/row ordering across prefill-to-decode transfer, not metadata lifetime
  or shape.

C64 correction applied:

- `_free_decode_metadata_index_if_held` now clears `Req.output_topk_p`,
  `Req.output_topk_index`, and `Req.hidden_states_tensor` after returning the
  metadata slot to the allocator.
- The transfer hot path still stores views and does not clone; the request owns
  the slot until `process_prebuilt` has synchronously consumed the tensors.
- Unit expectations were extended so future changes cannot reintroduce stale
  reusable-slot aliases on long-lived decode requests.

C64 verification note:

- Local `py_compile` for the touched Python modules passed.
- Local pytest collection for `test_decode_queue_compaction.py` is blocked in
  this environment by missing `orjson`; use the g0034 container for this test.

### C66. 2026-05-30 metadata checksum debug guard compatibility

Checked fact:

- The remote decode queue compaction test uses a minimal fake `spec_algorithm`
  with `is_none()` but without `is_eagle()`.  The production decode transfer
  path already gates EAGLE metadata ownership on `not spec_algorithm.is_none()`;
  the new checksum-only debug branch used the narrower `is_eagle()` method and
  therefore broke this test double without changing production semantics.

Correction:

- The `[EAGLE_ACCEPT_DEBUG][metadata_get]` guard now follows the existing
  transfer ownership contract: emit only when speculative decoding is active
  (`not spec_algorithm.is_none()`) and the debug sampler allows it.
- This is debug-only; it does not change metadata slot lifetime or request
  scheduling behavior.

Verification:

- Local `py_compile` passed for `decode.py`, the metadata release helper file,
  and the decode queue compaction test.
- Remote container verification on g0034 passed:
  `PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py`
  -> `11 passed, 5 warnings`.

### C67. 2026-05-30 exact non-page CP cache hits still expose sub-page tails

Finding:

- C16/C54 floor CP radix hits when a request extends beyond an existing
  non-page-aligned valid-tail node, but an exact key hit still exposes the full
  token-precise tail.
- In normal prefill scheduling this is still an extend: the radix key is capped at
  `input_len - 1`, so an exact EAGLE/bigram key hit with length `N` means the
  model will compute at least the next current token while the CP shared-KV
  backend sees `extend_prefix_len=N`.
- If `N` is not page-aligned, CP shared-KV current reuse either cannot use the
  page-aligned prefix contract or must enter a mostly-zero/tiny split shape.
  This matches the recurring symptom pattern where repeated cache-hit requests
  are page-tail sized and EAGLE accept length collapses without transfer errors.

Decision:

- Treat CP HiCache radix cache as page-granular for scheduler-visible hits: an
  exact non-page valid-tail hit is floored to the previous page boundary, just
  like the extension case.
- The sub-page tail is sacrificed and recomputed by the new request. This costs
  at most one page of reusable prefix, but keeps `prefix_indices`,
  `cache_protected_len`, prepared backup start, and CP shared-KV materialization
  on the same page-aligned contract.

Regression first:

- Added failing tests on g0034 before production changes:
  - `test_cp_match_prefix_floors_exact_valid_tail_for_exact_key`
  - `test_cp_insert_replaces_exact_valid_tail_from_page_boundary`
- Both failed under the old helper: match exposed `[0..5]` and insert reported
  `prefix_len=6` for a 6-token node on 4-token pages.

C67 implementation update:

- Added `MatchPrefixParams.cp_floor_exact` so scheduler-visible CP prefix matches can
  floor exact non-page valid-tail hits, while the internal `cache_unfinished_req()`
  post-insert refresh can keep its own just-inserted exact tail and avoid
  invalidating current-request accounting.
- `HiRadixCache._cp_floor_exact_valid_tail_extension_len()` now floors exact
  non-page hits when `cp_floor_exact=True` or when the request key extends beyond
  the child.  Prepared-backup insertion enables the same exact-key floor so the
  recomputed page-tail backup attaches to the replacement suffix node.
- `RadixCache.cache_unfinished_req()` calls its internal refresh match with
  `cp_floor_exact=False`; this preserves the previous exact-duplicate accounting
  fix for repeated tiny EAGLE requests.
- Existing CP host-hit expectations were updated from token-granular `6` to
  page-granular `4` for a 6-token backed node on 4-token pages.

C67 verification:

```text
remote g0034 container:
  PYTHONPATH=python python -m pytest -q \
    test/registered/unit/mem_cache/test_cp_hicache_metadata.py \
    test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py
  -> 109 passed, 5 warnings
```

### C68. 2026-05-30 accept-length collapse correlates with draft partial-current reuse

Finding:

- Latest available remote process is no longer running; `/mnt/beegfs/cjy/log` still contains the previous run as the newest evidence:
  - Prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_122138.log`
  - Decode: `/mnt/beegfs/cjy/log/decode0_20260530_122148.log`, `/mnt/beegfs/cjy/log/decode1_20260530_122154.log`
- Compared with the earlier baseline-ish run (`20260529_1605xx`), the failing run differs on the EAGLE/NextN draft MLA path:
  - Earlier prefill `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_160555.log`: `draft_partial_current_reuse_disabled=64`, `branch=partial_current_sync=0`.
  - Failing prefill `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_122138.log`: `draft_partial_current_reuse_disabled=0`, `branch=partial_current_sync=112`, `branch=current_only=48`.
- Decode accept distribution regressed at the same time:
  - `decode0_20260529_160608.log`: `n=502`, `avg=1.324`, `median=1.18`, `last100=1.424`, `zero=0`.
  - `decode1_20260529_160611.log`: `n=468`, `avg=1.348`, `median=1.20`, `last100=1.351`, `zero=0`.
  - `decode0_20260530_122148.log`: `n=167806`, `avg=0.068`, `median=0.0`, `last100=0.01`, `zero=158223`.
  - `decode1_20260530_122154.log`: `n=160622`, `avg=0.068`, `median=0.0`, `last100=0.011`, `zero=151281`.
- The failing logs predate the newest `metadata_set/get/prebuilt_state` checksum probes, so metadata handoff values are still unverified in a fresh process.

Working hypothesis:

- Target-model partial current reuse can remain enabled, but EAGLE/NextN draft partial current reuse is not yet a validated correctness contract.
- For draft cache-hit suffixes, the safer contract is the previous full-materialize path: direct write the local draft suffix into the draft pool, then materialize the full prefix+suffix from the pool/all-rank compose.  This is slower than partial splice but matched the earlier non-collapsed accept distribution.

Planned correction:

- Keep draft current-only reuse enabled for empty-prefix batches.
- Disable only draft partial-current splice (`prefix_len > 0`) and emit an explicit `[CP_SHARED_KV_FALLBACK][current_reuse] reason=draft_partial_current_reuse_disabled` warning when this gate fires.
- Do not add new collectives; this restores the existing full-materialize path rather than inventing a new synchronization path.

C68 implementation update:

- `should_reuse_current_extend_kv()` now returns `True` immediately for current-only batches, preserving draft and target empty-prefix reuse.
- For partial-current batches, EAGLE/NextN draft inputs (`spec_info.is_draft_input()`) now return `False` and emit the explicit fallback warning:
  `[CP_SHARED_KV_FALLBACK][current_reuse] reason=draft_partial_current_reuse_disabled`.
- Target partial-current reuse remains enabled.

C68 verification:

```text
remote g0034 container:
  PYTHONPATH=python python -m pytest -q \
    test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_should_reuse_current_extend_kv_disables_draft_partial_cache_hit_suffix
  -> 1 passed

  PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py
  -> 73 passed, 5 warnings, 2 subtests passed
```

Runtime validation still needed:

- Start a fresh prefill/decode process with the synced code and `SGLANG_EAGLE_ACCEPT_DEBUG=1`.
- Expected prefill evidence: draft cache-hit suffix rows should log `draft_partial_current_reuse_disabled`; draft `branch=partial_current_sync` should disappear except for old logs.
- Expected decode evidence: accept distribution should move away from the `avg≈0.068 / median=0` collapse if C68 is the active root cause.

### C69. 2026-05-30 fresh remote run: parameters correct, draft partial splice disabled, accept still collapsed

Checked runtime:

- Prefill log: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_143043.log`.
- Decode logs: `/mnt/beegfs/cjy/log/decode0_20260530_143046.log`, `/mnt/beegfs/cjy/log/decode1_20260530_143049.log`.
- Prefill process args include `--enable-nsa-prefill-cp-shared-kv`, `--attn-cp-size 8`, `--enable-hierarchical-cache`, `--hicache-size 150`, `--hicache-io-backend direct`, and `--hicache-mem-layout page_first_direct`.
- Prefill env includes `SGLANG_CP_DRAFT_SHARED_KV=1`, `SGLANG_CP_SHARED_KV_CURRENT_REUSE=1`, `SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH=1`, `SGLANG_CP_SHARED_KV_FUSED_MLA_STORE=1`, `SGLANG_CP_SHARED_KV_FUSED_INDEX_MQA_PREPARE=1`, `SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1`, and `SGLANG_EAGLE_ACCEPT_DEBUG=1`.

Findings:

- C68 gate is active: prefill marker counts are `draft_partial_current_reuse_disabled=64`, `branch=partial_current_sync=0`, `branch=current_only=64`, `metadata_set=128`, `cp_draft_hidden=256`.
- Draft cache-hit suffixes now use `branch=full_materialize` with `can_current_reuse=False`, as intended by C68.
- EAGLE handoff metadata is non-zero after warmup and decode `metadata_get` matches `prebuilt_state` for sampled rows; the prebuilt-state copy path is not obviously corrupting metadata.
- Accept length did **not** recover:
  - `decode0_20260530_143046.log`: `n=34784`, `avg=0.075`, `median=0.0`, `last100=0.235`, `zero=32516`.
  - `decode1_20260530_143049.log`: `n=30638`, `avg=0.060`, `median=0.0`, `last100=0.082`, `zero=29006`.

Conclusion:

- C68 restored the intended draft full-materialize path but is not the accept-collapse root cause by itself.
- The next likely surfaces are draft KV content/row-order correctness during prefill-to-decode transfer, or a decode-side EAGLE interpretation issue after `prebuilt_state` receives valid non-zero metadata.

### C70. 2026-05-30 accept collapse root-cause candidate: TAI current-slot fill assumes contiguous logical page ids

Finding:

- After the page-granular/no-short-fallback change, the hot target MLA partial-current path uses `fill_current_kv_page_slots_and_remap_locs()` to place current suffix KV into page-slot dense buffers.
- With `SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1`, this dispatches to `tai_kernel.nsa_prefill.fill_current_token_kv_page_slots_and_remap_locs`.
- The TAI masking kernel `_mask_current_page_slack_locs_direct_kernel` treats all pages between `first_current_page` and `last_current_page` as current pages and derives `candidate_current_row` by page ordinal:
  `candidate = (logical_page - first_current_page) * page_size + token_offset - first_current_offset`.
- CP/paged KV allocator locs are page-based cache-pool addresses; current suffix pages are not guaranteed to have contiguous logical page ids. A suffix can use pages `[5, 25]` while unrelated prefix/query locs include page `10`.

Remote proof on g0034 container:

```text
page_size=4
page_inverse: logical page 5 -> dense page 1, page 10 -> dense page 2, page 25 -> dense page 3
logical_locs=[20,21,40,41,100,101,102,103]
current_locs=[20,21,100,101]
TAI mixed_locs=[[4, 5, -1, -1, -1, -1, -1, -1]]
TAI current_mask=[[True, True, False, False, False, False, False, False]]
expected mixed_locs should preserve unrelated prefix page 10 and mark current page 25 rows:
  [[4, 5, 8, 9, 12, 13, -1, -1]]
expected current_mask:
  [[True, True, False, False, True, True, False, False]]
```

Impact:

- This directly explains why accept length collapses while metadata transfer still looks valid: target prefill attention can silently hide valid prefix/current KV rows in the partial-current path, producing wrong next-token/hidden handoff for EAGLE.
- The bug is not in the Python torch reference: Python uses exact page membership (`torch.unique(current_pages)`) and exact current-loc membership. The bug is the TAI kernel's contiguous-page assumption.
- This is consistent with the regression appearing after the page-slot current-compose path became hot and TAI materialize was enabled.

Next fix:

- Fix TAI `fill_current_token_kv_page_slots_and_remap_locs` so current/slack detection uses exact `page_inverse` page membership plus exact `current_locs` membership, or a search over `current_locs`, not a contiguous page interval.
- Add a CUDA unit test with non-contiguous current pages and an unrelated prefix page between them.
- Until that test passes, do not use this TAI kernel as evidence that page-aligned CP shared-KV semantics are correct.

C70 implementation update:

- `tai-kernel` now builds exact dense-row and dense-page masks from `current_locs` after remapping them through `page_inverse`.
- The slack-mask kernel no longer infers current membership from a contiguous logical page interval. It now uses:
  - `current_row_mask[dense_row]` for exact current-token membership;
  - `current_page_mask[dense_page]` for page-slack masking.
- This preserves unrelated prefix pages whose logical page ids fall numerically between two sparse current suffix pages.

C70 verification:

```text
remote g0034 container, /mnt/beegfs/cjy/tai-kernel:
  PYTHONPATH=python python -m pytest -q \
    tests/nsa_prefill/test_cp_shared_kv_materialize.py::test_fill_current_token_kv_page_slots_handles_sparse_current_pages -q
  -> passed

  PYTHONPATH=python python -m pytest -q tests/nsa_prefill/test_cp_shared_kv_materialize.py
  -> 10 passed

runtime import check:
  tai_kernel.nsa_prefill.cp_shared_kv_materialize.__file__
  -> /mnt/beegfs/cjy/tai-kernel/python/tai_kernel/nsa_prefill/cp_shared_kv_materialize.py
  hasattr(..., "_zero_current_dense_masks_kernel") -> True
```

Runtime validation still needed:

- Restart prefill/decode with the updated `tai-kernel` and the same `SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1` setting.
- Expected prefill evidence: target `forward_partial_current_reuse` can remain active, but accept length should no longer collapse to `avg≈0.06 / median=0` if C70 was the dominant root cause.

### C71. 2026-05-31 post-C70 runtime observation: aggregate decode accept recovered, sampled EAGLE rows still zero-heavy

Runtime observed:

- Prefill process on `g0034` started at `2026-05-30 23:52:29 CST` with:
  - `SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1`
  - `SGLANG_CP_SHARED_KV_CURRENT_REUSE=1`
  - `SGLANG_EAGLE_ACCEPT_DEBUG=1`
  - `--enable-nsa-prefill-cp-shared-kv`
  - `--hicache-io-backend direct --hicache-mem-layout page_first_direct`
- The synced TAI file timestamp is `2026-05-30 22:57:41 CST`, before this restart.
- Container import resolves to `/mnt/beegfs/cjy/tai-kernel/python/tai_kernel/nsa_prefill/cp_shared_kv_materialize.py` and exposes `_zero_current_dense_masks_kernel`, so the new process should be using the C70 fixed module path.

Fresh log evidence at `2026-05-31 00:00-00:01 CST`:

```text
current decode0_20260530_155232.log:
  Decode batch accept: n=123 avg=2.2476 median=2.25 p10=1.93 p90=2.52 zero=0 last100=2.2534
  EAGLE_ACCEPT_DEBUG avg_accept: n=1875 avg=0.1229 median=0 zero=1657 last100=0.2900
  sampled accepted_draft: n=1875 avg=0.0245 median=0 zero=1848 last100=0.0000 output_zero=0

current decode1_20260530_155236.log:
  Decode batch accept: n=118 avg=2.1740 median=2.15 p10=1.88 p90=2.45 zero=0 last100=2.1797
  EAGLE_ACCEPT_DEBUG avg_accept: n=1903 avg=0.0851 median=0 zero=1741 last100=0.0300
  sampled accepted_draft: n=1903 avg=0.0231 median=0 zero=1880 last100=0.0100 output_zero=0

previous bad decode0_20260530_143046.log:
  Decode batch accept: n=1773 avg=1.3924 median=1.2 zero=0 last100=1.6584
  EAGLE_ACCEPT_DEBUG avg_accept: n=52797 avg=0.0548 median=0 zero=48785 last100=0.0300

previous bad decode1_20260530_143049.log:
  Decode batch accept: n=1729 avg=1.4446 median=1.3 zero=0 last100=1.7419
  EAGLE_ACCEPT_DEBUG avg_accept: n=49015 avg=0.0666 median=0 zero=44342 last100=0.0000
```

Interpretation:

- The primary aggregate `Decode batch, accept len` signal has recovered above both the immediate bad run and the older no-debug baseline-ish run (`~1.3`). It is not showing the severe accept-collapse pattern.
- No `Traceback`, `Health check failed`, `detokenizer hang`, `KVCapacityWaitError`, or output-zero sample was found in the current prefill/decode logs during this observation window.
- The per-step `EAGLE_ACCEPT_DEBUG` sample remains zero-heavy because it reports sampled rows/steps, not the same aggregate as `Decode batch accept len`. Treat it as secondary evidence; do not use it alone to claim accept collapse while aggregate decode accept remains healthy.
- Remaining visible fallbacks are still present and should be cleaned separately:
  - `prefix_not_page_aligned` during zero-prefix/early requests;
  - `draft_partial_current_reuse_disabled` by the current draft safety gate.

Status:

- C70 is supported by runtime evidence as the dominant cause of the previous aggregate accept-length regression.
- Continue longer ETE observation before calling the whole run stable, especially under host-cache-full pressure.

C71 continued observation at `2026-05-31 00:03 CST`:

```text
current decode0_20260530_155232.log:
  Decode batch accept: n=307 avg=2.2528 median=2.25 p10=1.93 p90=2.55 zero=0 last100=2.2494
  EAGLE_ACCEPT_DEBUG avg_accept: n=4828 avg=0.1984 median=0 zero=3902 last100=0.7492
  sampled accepted_draft output_zero=0

current decode1_20260530_155236.log:
  Decode batch accept: n=309 avg=2.2042 median=2.20 p10=1.90 p90=2.52 zero=0 last100=2.2055
  EAGLE_ACCEPT_DEBUG avg_accept: n=5021 avg=0.1782 median=0 zero=4152 last100=0.3650
  sampled accepted_draft output_zero=0
```

No new runtime error was found in the current logs in this observation window except warmup-time max-m reduction warnings.

### C72. 2026-05-31 second runtime log recheck: accept remains stable after longer run

Observed at `2026-05-31 00:17 CST`:

- Prefill `g0034` and decode `g0035/g0036` launch_server processes are still alive.
- Latest logs remain:
  - `sglang_cp_hicache_20260530_155230.log`
  - `decode0_20260530_155232.log`
  - `decode1_20260530_155236.log`

Fresh decode aggregate accept evidence:

```text
decode0_20260530_155232.log:
  Decode batch accept: n=1228 avg=2.2488 median=2.23 p10=1.93 p90=2.58 zero=0 last100=2.1639
  EAGLE_ACCEPT_DEBUG avg_accept: n=18811 avg=0.2070 median=0 p90=1 zero=14988 last100=0.0200
  sampled accepted_draft output_zero=0

decode1_20260530_155236.log:
  Decode batch accept: n=1211 avg=2.2405 median=2.23 p10=1.93 p90=2.57 zero=0 last100=2.2233
  EAGLE_ACCEPT_DEBUG avg_accept: n=18480 avg=0.1854 median=0 p90=1 zero=15160 last100=0.0700
  sampled accepted_draft output_zero=0
```

Error scan:

- No `Traceback`, `Exception`, `ERROR`, `CRITICAL`, `Health check failed`, `detokenizer hang`, `KVCapacityWaitError`, request invalidation, or completion/output-zero signature was found in the current observation window.
- Only warmup-time max-m reduction warnings matched the broad error grep.

Prefill marker counts:

```text
forward_partial_current_reuse 7984
forward_partial_current_sync_compose 1968
branch=full_materialize 7984
branch=partial_current_sync 0
branch=current_only 56
draft_partial_current_reuse_disabled 64
CP_SHARED_KV_FALLBACK 208
prefix_not_page_aligned 144
owner-lane evict END 3144
_evict_for_compute_owner_lanes 6288
KVCapacityWaitError 0
Traceback 0
Health check failed 0
detokenizer hang 0
```

Interpretation:

- Aggregate decode accept remains stable around `2.24` after a longer run, so C70 remains supported as the accept-collapse fix.
- Device owner-lane evictions are occurring frequently but not causing OOM/hang in this window.
- Remaining visible work is performance/fallback cleanup, not an immediate correctness crash in this run.

### C73. 2026-05-31 current fallback log classification

Current runtime `sglang_cp_hicache_20260530_155230.log` fallback reasons:

```text
120 x [CP_SHARED_KV_FALLBACK][mla_prefetch]   reason=prefix_not_page_aligned prefix_len=0 page_size=64
120 x [CP_SHARED_KV_FALLBACK][index_prefetch] reason=prefix_not_page_aligned prefix_len=0 page_size=64
 64 x [CP_SHARED_KV_FALLBACK][current_reuse]  reason=draft_partial_current_reuse_disabled prefix_lens=[320] ...
```

Interpretation:

- `prefix_not_page_aligned` is currently all `prefix_len=0`. It is a prefetch optimization skip for current-only/no-cache-prefix batches, not a non-page-tail cache-hit fallback. Since there is no host/L1 prefix to prefetch, creating MLA/index prefetcher returns `None`. The warning label is misleading for this case; it should likely be classified as `create_skip/no_prefix` or gated before constructing prefetchers.
- `draft_partial_current_reuse_disabled` is the deliberate C68 EAGLE/NextN safety gate. Draft cache-hit partial-current splice is disabled, so draft uses full materialize while target partial-current reuse remains active.

Impact:

- These current fallback logs do not indicate the old page-tail fallback path reappearing.
- They are still noisy and should be cleaned/classified before finalizing logs, but they are not the current accept-length correctness regression.

### C74. 2026-05-31 L2->L1 HiCache load-back async check

Question: whether current L2(host)->L1(device) HiCache load-back/prefetch is truly async.

Code evidence:

- `schedule_policy.py` calls `tree_cache.init_load_back()` when `req.host_hit_length > 0`; this prepares load-back metadata and queues load operations, but does not itself wait for all KV bytes to land.
- `scheduler.py` calls `tree_cache.ready_to_load_host_cache()` before `prepare_for_extend()`; this maps to `HiCacheController.start_loading()` and returns `hicache_consumer_index` to the batch.
- `HiCacheController.start_loading()`:
  - merges pending load ops;
  - records a producer start event;
  - enters `self.load_stream`;
  - calls `mem_pool_host.load_to_device_per_layer(...)` for each layer;
  - records one per-layer event via `producer_event.complete(i)`;
  - appends an ack with the final per-layer event.
- Device KV pool access waits only on the needed layer event:
  - normal `get_key_buffer()/get_value_buffer()` waits via `LayerDoneCounter.wait_until(layer_id)`;
  - prefetch getter `get_key_buffer_for_prefetch(..., stream)` waits the same event on the prefetch stream using `wait_until_on_stream`.
- With current launch `--hicache-io-backend direct --hicache-mem-layout page_first_direct`, `NSATokenToKVPoolHost.load_to_device_per_layer()` calls `tai_kernel.nsa_prefill.transfer_kv_per_layer_direct_pf_lf` for MLA KV and indexer buffers.
- The TAI H2D op uses `cudaMemcpyBatchAsync` on `at::cuda::getCurrentCUDAStream(device_id)`, so when called under `self.load_stream` it enqueues copy-engine work on the load stream and returns without CPU-side copy completion synchronization.

Runtime evidence from current log:

```text
[HiCache-load] load_back CP:         5424
[HiCache-load] load_back CP SUCCESS: 5424
CP_HICACHE_FALLBACK][cp_load_back:   0
```

Interpretation:

- The bulk L2->L1 KV byte transfer is async with respect to model compute: it is queued on a separate `load_stream`, and model streams wait per layer only when consuming that layer.
- It is not a fully zero-cost async path. There is still synchronous CPU preparation before the async H2D copies are enqueued:
  - `load_cp()` does owner-lane planning/allocation/metadata concatenation;
  - direct mode moves `device_indices` to CPU before enqueue;
  - the TAI op builds CPU-side `cudaMemcpyBatchAsync` pointer/size vectors per layer.
- Current design is therefore best described as: async DMA/copy-engine transfer with per-layer consume waits, but synchronous CPU submit/preparation overhead. If performance still regresses on cache-hit loads, the remaining suspect is enqueue/planning overhead or insufficient lookahead, not blocking H2D memcpy completion.

### C75 — runtime L2→L1 prefetch/load-back async contract re-check

Finding date: 2026-05-31.

Current remote prefill process on g0034 was inspected via `/proc/<pid>/cmdline` and is currently launched with:

- `--hicache-io-backend kernel`
- `--hicache-mem-layout page_first`
- `SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH=1`
- `SGLANG_CP_SHARED_KV_FUSED_INDEX_MQA_PREPARE=1`
- `SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1`

This means current L2(host HiCache) → L1(device KV pool) load-back is **not** using the TAI `page_first_direct`/`cudaMemcpyBatchAsync` 0-SM copy path. It is using the SGLang `kernel + page_first` path:

- `HiRadixCache.load_back()` queues `CacheOperation` via `cache_controller.load_cp()`.
- `Scheduler._get_new_batch_prefill_raw()` calls `tree_cache.ready_to_load_host_cache()` before `prepare_for_extend()`.
- `HiCacheController.start_loading()` switches to `self.load_stream` and calls `mem_pool_host.load_to_device_per_layer()` per layer.
- `MLATokenToKVPoolHost.load_to_device_per_layer()` with `io_backend == "kernel"` and `layout == "page_first"` calls `sgl_kernel.kvcacheio.transfer_kv_per_layer_mla_pf_lf()`.
- `sgl-kernel/csrc/kvcacheio/transfer.cu` launches the transfer kernel on `at::cuda::getCurrentCUDAStream()`, which is `load_stream` in this context, and records per-layer completion events.
- Device KV consumers wait per layer through `LayerDoneCounter.wait_until()` / `wait_until_on_stream()` before reading normal KV or prefetch KV buffers.

Conclusion:

- It is **async in the CUDA stream sense**: load-back kernels are enqueued on a separate load stream, and forward/prefetch consumers wait only at per-layer consumption.
- It is **not async in the 0-SM/DMA-copy-engine sense** for the current launch: the `kernel + page_first` path runs an SM-consuming transfer CUDA kernel.
- The TAI direct path (`--hicache-io-backend direct --hicache-mem-layout page_first_direct`) would enqueue `cudaMemcpyBatchAsync` and fail fast if unavailable, but that is not the current runtime configuration.

Risk / action:

- If we expect L2→L1 load-back to avoid SM interference with attention/index/MLA kernels, current launch parameters must switch back to `direct + page_first_direct`, and remote logs should confirm no `CP_HICACHE_FALLBACK][cp_load_back` and the TAI direct path is actually installed/imported.
- Even with direct mode, CPU-side planning/submission remains synchronous; only bulk bytes are async.

### C76 — correction: C75 inspected a non-target remote process

Finding date: 2026-05-31.

Correction: C75's `g0034` process inspection matched a non-target/stale prefill process launched with `--hicache-io-backend kernel --hicache-mem-layout page_first`. The user's intended launch command for the active experiment is:

- `--hicache-io-backend direct`
- `--hicache-mem-layout page_first_direct`

Therefore C75 must not be used as evidence about the intended experiment's L2→L1 path.

For the intended launch, the code path is:

- `HiCacheController.start_loading()` enters `self.load_stream`.
- `NSATokenToKVPoolHost.load_to_device_per_layer()` / `MLATokenToKVPoolHost.load_to_device_per_layer()` with `io_backend == "direct"` and `layout == "page_first_direct"` call `_load_tai_transfer_kv_per_layer_direct_pf_lf()` for both MLA KV and NSA indexer buffers.
- `tai_kernel.nsa_prefill.kvcacheio.transfer_kv_per_layer_direct_pf_lf()` calls `torch.ops.tai_kernel_ops.transfer_kv_per_layer_direct_pf_lf`.
- The TAI C++ implementation uses `cudaMemcpyBatchAsync` on the current CUDA stream, which is `load_stream` under `start_loading()`, and fails fast if CUDA/runtime support is missing.

Conclusion for the intended launch:

- L2→L1 bulk transfer should be async and copy-engine based, not SM-transfer-kernel based.
- It still has synchronous CPU-side prep/submission overhead before the async copies are queued.
- Runtime verification must match the intended process/log, not a stale/non-target prefill process.

### C77 — L2→L1 load-back CPU overhead granularity and per-request conversion feasibility

Finding date: 2026-05-31.

Question: whether the remaining CPU-side L2→L1 load-back overheads can be moved from per-layer to per-request.

Current granularity:

- Per request / per cache-hit load-back:
  - radix/HiCache load planning: `HiRadixCache.init_load_back()` → `load_back()` → `_build_cp_load_back_plan()`;
  - owner-lane capacity check: `CPSharedPagedTokenToKVPoolAllocator.compute_owner_lane_stats(page_owners)`;
  - optional owner-lane eviction planning: `_plan_cp_load_back_owner_lane_evictions()` / `_evict_cp_load_back_owner_lanes()`;
  - owner-lane allocation: `alloc_pages_with_owners(page_owners)`;
  - host/device token-index concatenation and queueing: `CacheController.load_cp()`;
  - direct-mode `device_indices.cpu()` in `CacheController.move_indices()`.
- Per layer:
  - `HiCacheController.start_loading()` calls `load_to_device_per_layer()` once per target layer and draft layer;
  - NSA target direct/page_first_direct calls TAI H2D once for MLA KV and once for indexer for each layer;
  - TAI `transfer_kv_per_layer_direct_pf_lf()` rebuilds `std::vector` memcpy descriptors and calls `cudaMemcpyBatchAsync()` on every layer;
  - `NSATokenToKVPoolHost._get_indexer_page_indices()` recomputes host/device page indices inside every indexer layer load, although the result is layer-invariant.

Feasibility:

1. Safe/low-risk per-request conversion:
   - Precompute indexer page indices once per load op and pass/reuse them for all indexer layers.
   - Precompute reusable page-pair metadata (`host_page`, `dst_start`) once per load op so the per-layer TAI call does not redo token-index CPU conversion/validation/page extraction.
   - Preserve per-layer submit/events so forward overlap semantics remain unchanged.

2. Medium-risk conversion:
   - Add a TAI H2D API that accepts precomputed page-pair metadata and layer range, then internally enqueues one `cudaMemcpyBatchAsync` per layer and optionally records/returns per-layer completion ordering. This reduces Python/C++ descriptor preparation churn but must preserve per-layer readiness.

3. High-risk / not recommended as first step:
   - Collapse all layers into one giant all-layer `cudaMemcpyBatchAsync` and only signal a final completion event. This makes the submit path per-request, but destroys per-layer load/forward overlap because layer 0 would effectively wait behind all later-layer copies unless a more complex event scheme is added.

Conclusion:

- Yes, some per-layer CPU work can be moved to per-request without changing correctness: indexer page-index derivation and page-pair metadata preparation are the first targets.
- The actual per-layer async submit should initially remain per-layer to preserve overlap and existing `LayerDoneCounter` semantics.
- A full per-request H2D op is possible only if it still preserves per-layer completion visibility; otherwise it trades CPU overhead for lost overlap.

### C78 — implemented per-request indexer page-index preparation for L2→L1 load-back

Finding date: 2026-05-31.

Change implemented:

- `HostKVCache` now exposes optional load-op lifecycle hooks:
  - `begin_load_to_device_op(host_indices, device_indices, io_backend)`
  - `end_load_to_device_op()`
- `HiCacheController.start_loading()` calls these hooks once per merged target load op and once per merged draft load op, then preserves the existing per-layer `load_to_device_per_layer()` loop and per-layer completion events.
- `NSATokenToKVPoolHost.begin_load_to_device_op()` computes indexer host/device page indices once using `_get_indexer_page_indices()` and stores them in `_active_load_indexer_page_indices`.
- `NSATokenToKVPoolHost._load_indexer_to_device_per_layer()` reuses the prepared indices when present; direct calls without a begin hook still compute indices locally, preserving the old API behavior.

Important stream correction:

- The begin hook is called inside `with device_module.stream(self.load_stream)`, before the per-layer loop.
- This is intentional: in `kernel + page_first` mode the prepared page-index tensors can be CUDA tensors, so deriving them on the load stream avoids introducing a cross-stream producer/consumer race. In `direct + page_first_direct` mode the tensors are CPU, so this has no downside.

What this optimizes:

- It removes the per-layer repeated `reshape(... page_size)[:, 0] // page_size` indexer page-index derivation for load-back.
- It does not yet remove per-layer TAI `cudaMemcpyBatchAsync` descriptor construction for the actual KV/indexer copies.
- It preserves per-layer overlap and `LayerDoneCounter` semantics.

Verification:

- RED on remote before implementation:
  - new `test_nsa_indexer_load_reuses_precomputed_page_indices_across_layers` failed with missing `begin_load_to_device_op`;
  - new `test_start_loading_prepares_load_op_once_for_all_layers` failed because `start_loading()` did not call the begin hook before per-layer loads.
- GREEN after implementation on g0034 container:
  - `python -m py_compile python/sglang/srt/mem_cache/memory_pool_host.py python/sglang/srt/managers/cache_controller.py test/registered/unit/managers/test_hicache_controller_cp.py`
  - `PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py` → `61 passed, 3 warnings`.
  - `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_nsa_pool_host_unit.py::TestNSAIndexerPageIndices` → `4 passed, 3 warnings`.

Known verification gap:

- Full `test_nsa_pool_host_unit.py` exited with code 139 after the first CUDA test in the remote container. This is not proven related to the current Python load-op hook change; the CPU-only page-index subset passes. Do not claim the CUDA host-transfer suite is clean until this is isolated separately.

### C79 — 2026-05-30 full `test_nsa_pool_host_unit.py` CUDA exit 139 root cause

Finding: the CUDA 139 is a native segfault in the old `sgl_kernel.kvcacheio.transfer_kv_all_layer_direct_lf_pf` path, not a Python exception and not the new per-request H2D indexer-page precompute hook.

Evidence:
- Remote isolated run: `test_device_to_host_indexer_kernel_layer_first` passed, `test_device_to_host_indexer_direct_layer_first` passed, then `test_device_to_host_indexer_direct_page_first_direct` crashed with `Segmentation fault (core dumped)`.
- Fault stack enters `cuMemcpyBatchAsync_v2` -> `cudaMemcpyBatchAsync` -> `transfer_kv_page_first_direct_impl<true>` -> `transfer_kv_all_layer_direct_lf_pf`.
- Remote container reports `torch.version.cuda == 13.0`, so the CUDA 13 `cudaMemcpyBatchAsync` v2 ABI is active.
- `sgl-kernel/csrc/kvcacheio/transfer.cu` still casts `cudaMemcpyBatchAsync` to the pre-CUDA-13 signature with a `fail_idx` argument in `transfer_kv_page_first_direct_impl`, while `tai-kernel/python/tai_kernel/nsa_prefill/csrc/kvcacheio_lf_pf.cu` already uses a `CUDA_VERSION >= 13000` signature gate without `fail_idx`.
- Re-running the same NSA page-first-direct D2H path with `pin_memory=True` still exits 139, so pinned/pageable host allocation is not the root cause.

Impact:
- The failing unit covers the all-layer D2H backup indexer path for `direct + page_first_direct`; production per-layer async backup/load should prefer the TAI kernel path and avoid this stale sgl-kernel all-layer direct path on CUDA 13.
- Do not use this failing full suite as evidence that the Python H2D precompute change is wrong; the isolated Python/CPU indexer tests and `test_hicache_controller_cp.py` still pass.

Next correction target:
- Either route NSA page-first-direct indexer all-layer D2H backup away from `sgl_kernel.transfer_kv_all_layer_direct_lf_pf` on CUDA 13, or port the CUDA-13-safe `cudaMemcpyBatchAsync` ABI handling from tai-kernel into sgl-kernel. Since this branch intentionally avoids maintaining sgl-kernel for our path, the cleaner SGLang-side fix is to avoid the stale all-layer sgl-kernel op for NSA page-first-direct indexer backup.

### C80 — 2026-05-30 SGLang-side route away from sgl-kernel all-layer direct LF->PF

Decision: follow option 1 from C79. Do not patch `sgl-kernel` for this branch; route `direct + page_first_direct` all-layer D2H backup through the CUDA-13-safe TAI per-layer direct LF->PF op.

Implementation:
- `MHATokenToKVPoolHost.backup_from_device_all_layer(..., io_backend="direct")` now loops layers and calls `tai_kernel.nsa_prefill.transfer_kv_per_layer_direct_lf_pf` for K/V when `layout == "page_first_direct"`.
- `MLATokenToKVPoolHost.backup_from_device_all_layer(..., io_backend="direct")` now loops layers and calls the same TAI op for MLA KV when `layout == "page_first_direct"`.
- `NSATokenToKVPoolHost._backup_indexer_from_device_all_layer(..., io_backend="direct")` now loops layers and calls the same TAI op for NSA indexer pages with `page_size=1`.

Why MHA/MLA had to change too:
- Avoiding only the NSA indexer path was insufficient. `NSATokenToKVPoolHost.backup_from_device_all_layer()` first calls the MLA base backup, and the base MLA `page_first_direct` path still entered `sgl_kernel.transfer_kv_all_layer_direct_lf_pf`, causing the same CUDA 13 segfault before the indexer route ran.

Test corrections:
- `test_nsa_pool_host_unit.py` now has route tests that patch `transfer_kv_all_layer_direct_lf_pf` to fail if the stale sgl-kernel path is used for MHA, MLA, or NSA indexer page-first-direct all-layer backup.
- The CUDA transfer test synchronizes after direct backup before reading CPU host buffers because the TAI memcpy-batch path is asynchronous.
- The page-first-direct NSA indexer expected shape uses `[host_page, layer_id, 0]`; the host indexer buffer layout is `(page, layer, 1, stride)`, not `(page, layer, stride)`.

Verification on g0034 container:
- RED before implementation:
  - `TestPageFirstDirectAllLayerBackupRoute` failed because MHA/MLA still called `sgl_kernel.transfer_kv_all_layer_direct_lf_pf`.
  - Full `test_nsa_pool_host_unit.py` exited 139 in `cuMemcpyBatchAsync_v2` before this route change.
- GREEN after implementation:
  - `PYTHONPATH=python python -m pytest -q -s test/registered/unit/mem_cache/test_nsa_pool_host_unit.py` → `10 passed, 3 warnings`.
  - `PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py` → `61 passed, 3 warnings`.

Remaining risk:
- This makes the all-layer direct/page-first backup API submit one TAI per-layer memcpy-batch op instead of one stale sgl-kernel all-layer op. It is correct and CUDA-13-safe, but it can increase CPU submit count on all-layer backup paths. Production forward-overlap paths already operate per-layer, so this is acceptable for the current branch goal.

### C81 — 2026-05-31 correction: do not reopen draft partial-current reuse before fixing root cause

Correction:

- Draft partial-current reuse must not be reopened merely by deleting the
  `draft_partial_current_reuse_disabled` guard.
- The observed accept-length regression must be fixed first, then the default
  can be reopened without an env gate.
- Until that root cause is fixed and verified, the safe contract remains:
  draft current-only reuse is allowed, draft cache-hit partial-current reuse is
  disabled with the explicit `draft_partial_current_reuse_disabled` warning.

Current known facts:

- Draft HiCache L2(host)→L1(device) load-back exists via `CacheController.load_cp()`
  and `CacheController.start_loading()` using `draft_load_queue` and
  `draft_mem_pool_host.load_to_device_per_layer(...)` when `has_draft_hicache` is
  true.
- That L2→L1 load-back is separate from the CP shared-KV MLA/index next-layer
  prefetcher; the latter is still target-only because EAGLE/NextN has one
  executable draft layer.

Next target:

- Audit the draft partial-current compose path (`nsa_backend.py` plus
  `materialize_prefix_and_reuse_current_kv_page_slots`) for target/draft layout,
  layer-id, current-suffix placement, and page-slot mapping mismatches before
  reopening the guard.

### C82 — 2026-05-31 fixed-before-open path for draft partial-current reuse

Root-cause guard added before reopening draft partial-current reuse:

- The risky failure mode was not the draft predicate itself; it was accepting a
  stale TAI current-slot fill kernel whose masking logic assumed the current
  suffix covered a dense page range from first current page to last current page.
- For sparse current pages, that stale kernel could mask unrelated prefix pages
  as suffix slack, corrupting partial-current compose while tensor shapes stayed
  valid.
- `cp_shared_kv_runtime.py` now validates the installed TAI current-slot fill
  kernel once per CUDA device with a sparse-page self-test before using it. If
  the self-test fails, the TAI result is not used and the code falls back to the
  torch reference path, which preserves correctness.

After this capability fix:

- Draft cache-hit partial-current reuse is reopened directly, without an env
  gate.
- Draft next-layer MLA/index prefetcher creation remains disabled; draft has one
  executable layer, so a future draft prefetch optimization should use an
  explicit same-layer contract.

Verification:

- Remote RED before runtime helper existed:
  `test_tai_current_slot_fill_is_skipped_when_sparse_page_self_test_fails`
  failed with missing `_tai_current_slot_fill_supports_sparse_pages`.
- Remote GREEN after runtime helper:
  `test_tai_current_slot_fill_is_skipped_when_sparse_page_self_test_fails` → pass.
- Remote installed-kernel evidence:
  `test_tai_current_slot_fill_sparse_page_self_test_passes_on_installed_kernel`
  → pass.
- Remote tai-kernel sparse-page tests:
  `test_fill_current_token_kv_page_slots_handles_sparse_current_pages`,
  `test_fill_current_token_kv_page_slots_handles_nonzero_first_page_offset`, and
  `test_fill_current_token_kv_page_slots_masks_current_page_slack` → `3 passed`.

Remaining risk:

- Need ETE run with draft `branch=partial_current_sync` active to confirm decode
  aggregate accept remains healthy. The code-level stale-kernel corruption path
  is now blocked, but runtime traffic is still required before treating this as
  final performance/correctness validation.

### C83 — 2026-05-31 Mooncake P2P handshake receives HTTP traffic on port 17000

Observed remote log:

- `readString: too large length from socket: 3564087971010531152`
- `SocketHandShakePlugin: failed to receive handshake message, malformed json`

Root-cause evidence:

- `3564087971010531152 == 0x31762f2054534f50`, little-endian bytes are
  `POST /v1`.
- Tcpdump on g0034 captured the offending packet:
  `10.20.34.3:<ephemeral> > 10.20.32.34:17000` with HTTP payload
  `POST /v1/chat/completions HTTP/1.1` and `Host: 10.20.32.34:17000`.
- In the same run, Mooncake P2P handshake selected port `17000` for one prefill
  scheduler process:
  `Transfer Engine RPC using P2P handshake, listening on 10.20.32.34:17000`.
- The errors repeat every ~30 seconds and start before request traffic while the
  model is still loading, so this is not caused by HiCache per-layer backup
  payload corruption.

Current conclusion:

- This is a port/protocol collision: an external HTTP client/prober is sending
  OpenAI HTTP traffic to a Mooncake metadata/P2P handshake listener.
- It is noisy and can mask real transfer errors, but it is not evidence that
  HiCache write/read payloads are corrupted.

Decision:

- Do not keep the temporary SGLang-side Mooncake port-range override.
- The code change that set `MC_MAX_PRC_PORT=16999` by default was reverted on
  2026-05-31 at user request.
- If this noise matters operationally, fix the external endpoint/prober or set
  Mooncake `MC_MIN_PRC_PORT`/`MC_MAX_PRC_PORT` explicitly in the launch
  environment instead of silently changing SGLang defaults.

### C84 — 2026-05-31 draft partial-current reuse also requires valid-row, not padded-loc, MLA gating

Finding:

- Reopening draft partial-current reuse still was not enough when `out_cache_loc`
  was page padded. The MLA gate in `nsa_backend.py` required
  `k.shape[0] == forward_batch.out_cache_loc.numel()` and the same for
  `k_rope`.
- Under the page-aligned cache contract, `out_cache_loc` may cover the padded
  physical page extent while MLA projection tensors only contain the valid
  current suffix rows from `extend_seq_lens_cpu`.
- That strict equality silently forces cache-hit draft suffixes (for example
  `extend_len=65`, padded locs `128`) away from partial-current reuse and back
  toward full materialization.

Fix direction:

- Validate current MLA tensors against valid current rows, not padded loc count.
- Slice `k`, `k_rope`, and `out_cache_loc` to `extend_seq_lens_cpu[0]` before
  composing current rows. Padded tail locs are physical reservation/slack, not
  visible KV rows.
- Keep draft next-layer MLA/index prefetcher disabled; this is only restoring the
  synchronous partial-current compose path for draft cache-hit suffixes.

Verification added:

- RED on remote before implementation:
  `test_current_extend_kv_rows_for_reuse_accepts_padded_out_cache_loc` failed
  because `current_extend_kv_rows_for_reuse` did not exist.
- RED on remote before implementation:
  `test_mla_current_reuse_gate_accepts_padded_out_cache_loc` found the stale
  `k.shape[0] == forward_batch.out_cache_loc.numel()` gate.

Remaining runtime check:

- Remote GREEN after implementation:
  `test_should_reuse_current_extend_kv_enables_draft_partial_cache_hit_suffix`,
  `test_current_extend_kv_rows_for_reuse_accepts_padded_out_cache_loc`,
  `test_mla_current_reuse_gate_accepts_padded_out_cache_loc`,
  `test_tai_current_slot_fill_is_skipped_when_sparse_page_self_test_fails`, and
  `test_tai_current_slot_fill_sparse_page_self_test_passes_on_installed_kernel`
  → `5 passed`.
- Remote broader runtime helper suite:
  `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py`
  → `81 passed, 5 warnings, 2 subtests passed`.
- Still requires user-driven ETE traffic to verify that draft `partial_current_sync`
  appears without accept-length collapse.

### C85 — 2026-05-31 CP HiCache eviction path and overhead findings

Code path checked:

- Extend allocation under CP shared KV uses compute-owner page allocation in
  `alloc_paged_token_slots_extend()`.
- If owner-lane allocation fails, `_evict_for_compute_owner_lanes()` computes
  per-owner deficits and calls `tree_cache.evict(EvictParams(...,
  owner_lane_deficits=...))`.
- CP HiCache routes that to `_evict_cp_owner_lane_deficit_nodes()`, which plans
  victim nodes by owner-lane contribution and frees backed GPU nodes through
  `_evict_backuped()` -> `CacheController.evict_device()` ->
  `mem_pool_device_allocator.free()`.
- Load-back can also evict device nodes through `_evict_cp_load_back_owner_lanes()`.
- Host write admission evicts host nodes through
  `_reserve_write_cp_indices_no_collective()` ->
  `_evict_cp_host_for_write_admission()` -> `CacheController.evict_cp_host()`.

Remote log evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_184620.log`:

- `MemCache-evict _evict_for_compute_owner_lanes`: 3224 attempt lines across 8
  ranks, about 403 logical eviction cycles in ~18 minutes.
- Average owner-lane request was ~18.5k tokens; average actual evicted was ~38k
  tokens, i.e. about ~19.5k token overshoot per rank-line.
- Small requests were common: 1336 owner-lane evict lines requested less than
  4096 tokens, but eviction can still remove an entire victim node.
- Host write-admission eviction appeared too: 1088 lines, about 136 logical
  host eviction cycles. Average local host freed was ~4.7k token slots.

Current conclusion:

- Eviction does not copy KV for already-backed nodes; it mainly frees allocator
  slots and updates radix/HiCache metadata. It should not consume PCIe bandwidth.
- It can still be CPU/GPU-scheduler expensive because it is synchronous on the
  scheduler path, scans/heapifies evictable leaves, emits many INFO logs, and
  `PagedTokenToKVPoolAllocator.free()` uses `torch.unique(free_index // page_size)`
  per victim.
- Per-owner capacity stats also do CUDA tensor reductions with `.item()` per
  owner lane, so failed owner-lane allocations can introduce host syncs.
- The large overshoot is structural: eviction frees whole radix nodes, not the
  exact deficit. This can cause L1 churn even after removing the older `* cp_size`
  multiplier.

Candidate optimizations to consider:

1. Rate-limit or demote hot-path INFO eviction logs; current log volume alone is
   large enough to affect performance during heavy churn.
2. Add a page-aligned fast path in `PagedTokenToKVPoolAllocator.free()` to use
   `free_index[::page_size] // page_size` when inputs are page-shaped, avoiding
   `torch.unique` for normal HiCache node eviction.
3. Cache/maintain owner-lane free counts incrementally in the CP allocator to
   avoid repeated `.sum().item()` reductions during owner-lane admission.
4. Improve victim selection to reduce overshoot for small deficits, or keep a
   small-node victim pool for small owner-lane shortages.

### C86 — 2026-05-31 Eviction hot-path logs demoted to DEBUG

User decision:

- Keep failure/fallback logs visible.
- Demote successful/no-op eviction hot-path diagnostics from INFO to DEBUG.

Implemented scope:

- `mem_cache/common.py`
  - `evict_from_tree_cache()` capacity-trigger message is DEBUG.
  - `_evict_for_compute_owner_lanes()` no-evictable, attempt, and result messages are DEBUG.
- `mem_cache/hiradix_cache.py`
  - CP load-back owner-lane eviction summary is DEBUG.
  - CP owner-lane no-victim / stale-victim / END summaries are DEBUG.
  - Deterministic CP host eviction success summary is DEBUG.
  - Generic `evict START/END`, `_evict_backuped`, `_evict_regular`, and host-slot eviction start are DEBUG.
  - `write_backup CP retry after deterministic host eviction` is DEBUG.
  - `write_backup CP FAILED after deterministic retry` is promoted to WARNING so true failures remain visible.

Not changed:

- Existing `warning`/`error` fallback, capacity failure, OOM, pin-release, and plan-divergence logs remain visible.
- Eviction semantics are unchanged; this only changes log levels.

Verification:

- Added `TestHiCacheEvictLoggingLevels::test_evict_hot_path_success_logs_are_debug_only`.
- RED on remote before implementation: failed because `MemCache-evict` success-path logs still used `logger.info`.
- GREEN on remote after implementation:
  `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py::TestHiCacheEvictLoggingLevels::test_evict_hot_path_success_logs_are_debug_only`
  → `1 passed`.
- Local and remote `py_compile` passed for touched Python files.

### C87 — 2026-05-31 all-gather owner-packed path must unpack to logical dense final product

Decision:

- Rank-major/owner-major packed pages are only a communication intermediate for
  the proposed all-gather replacement of dense all-reduce.
- The consumer-facing final product remains a dense buffer in request
  logical/page-table order.  Dense slot `i` corresponds to
  `logical_pages.reshape(-1)[i]`.
- The no-dummy final contract maps invalid/sentinel pages and invalid locs to
  `-1`; page 0 is not used as a silent dummy destination in the new final
  remap contract.

Implication for benchmarking:

- Measuring only `pack -> all_gather -> rank-major buffer` is incomplete and
  overestimates the all-gather approach.
- The production-style benchmark must include the reorder/unpack cost:
  `pack owner pages -> all_gather rank-major pages -> unpack to logical dense`.
- MLA remap must also be measured against the final no-dummy logical dense
  layout, not against the rank-major intermediate.

Implemented TAI support:

- `build_logical_dense_page_inverse()` builds a no-dummy zero-based
  logical-page-to-dense-slot inverse.
- `unpack_rank_major_pages_to_logical_dense()` copies the rank-major gathered
  intermediate back to logical dense order and zero-fills invalid/sentinel slots.

Verification:

- RED before implementation: the two new TAI CUDA tests failed with missing
  imports for `build_logical_dense_page_inverse` and
  `unpack_rank_major_pages_to_logical_dense`.
- GREEN after implementation on g0034 container:
  `test_build_logical_dense_page_inverse_remaps_without_dummy_page` and
  `test_unpack_rank_major_pages_to_logical_dense_matches_logical_order_reference`
  passed.

### C88 — 2026-05-31 owner-packed all-gather benchmark after logical-dense unpack

Remote benchmark environment:

- Host/container: g0034 / `sglang-glm5-dev-2`.
- Pattern: `--logical-pattern zigzag-owners`, 8 CP ranks, correctness checks enabled.
- Logs:
  - `/mnt/beegfs/cjy/log/cp_shared_kv_collective_bench_20260531_154608.log`
  - `/mnt/beegfs/cjy/log/cp_shared_kv_collective_bench_mla_focus_20260531_154720.log`
  - `/mnt/beegfs/cjy/log/cp_shared_kv_collective_bench_index_focus_20260531_154804.log`

MLA focused run (`repeat=30`, p50 GPU ms):

| tokens | dense all-reduce | pack+all-gather+unpack | dense+remap | gather+unpack+remap |
| ---: | ---: | ---: | ---: | ---: |
| 4096 | 0.148 | 0.153 | 0.150 | 0.184 |
| 8192 | 0.167 | 0.151 | 0.169 | 0.182 |
| 16384 | 0.211 | 0.171 | 0.216 | 0.182 |
| 40320 | 0.330 | 0.275 | 0.334 | 0.284 |

MLA breakdown, avg GPU ms:

- rank-major metadata: ~0.059–0.060 ms.
- rank-major unpack to logical dense: ~0.056 ms up to 16k, ~0.070 ms at 40k.
- cached logical-dense loc remap: ~0.041 ms.
- pack+all-gather intermediate grows with bytes: ~0.132 ms at 4k, ~0.249 ms at 40k.

Conclusion for MLA:

- After including the required logical-dense unpack, owner-packed all-gather is
  not a clear win for very short prefixes.  At 4k it is roughly equal/slower,
  especially when remap is included.
- It becomes beneficial around 16k+ in this microbenchmark and remains beneficial
  at 40k.
- The fixed overheads (metadata, unpack, remap) are large enough that this path
  should be gated by payload/size and should remain per-request, not per-layer.

Index focused run (`repeat=30`, p50 GPU ms):

| tokens | dense all-reduce | pack+all-gather+unpack |
| ---: | ---: | ---: |
| 4096 | 0.111 | 0.144 |
| 8192 | 0.110 | 0.142 |
| 16384 | 0.119 | 0.142 |
| 40320 | 0.125 | 0.140 |

Conclusion for index:

- The index payload is too small for owner-packed all-gather plus unpack to win
  in this benchmark. Dense all-reduce p50 remains faster.
- Do not switch index prefetch/materialize collective to owner-packed all-gather
  based on the current evidence.

### C89 — 2026-05-31 extended prefix sweep 4k–120k

Remote benchmark logs:

- MLA: `/mnt/beegfs/cjy/log/cp_shared_kv_collective_bench_mla_4k_120k_20260531_160744.log`
- Index: `/mnt/beegfs/cjy/log/cp_shared_kv_collective_bench_index_4k_120k_20260531_160839.log`

Configuration:

- 8 CP ranks, `--logical-pattern zigzag-owners`, `--warmup 8`, `--repeat 30`.
- Prefix tokens: `4096 8192 16384 32768 65536 98304 122880`.
- Final product for owner-packed path includes the required
  rank-major-to-logical-dense unpack.

MLA p50 GPU ms:

| tokens | pages | dense all-reduce | gather+unpack | ratio | dense+remap | gather+unpack+remap | ratio |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 4096 | 64 | 0.142 | 0.154 | 1.09 | 0.147 | 0.183 | 1.25 |
| 8192 | 128 | 0.166 | 0.151 | 0.91 | 0.170 | 0.182 | 1.07 |
| 16384 | 256 | 0.211 | 0.171 | 0.81 | 0.215 | 0.184 | 0.86 |
| 32768 | 512 | 0.288 | 0.243 | 0.84 | 0.290 | 0.250 | 0.86 |
| 65536 | 1024 | 0.446 | 0.378 | 0.85 | 0.450 | 0.385 | 0.85 |
| 98304 | 1536 | 0.607 | 0.519 | 0.86 | 0.611 | 0.525 | 0.86 |
| 122880 | 1920 | 0.726 | 0.627 | 0.86 | 0.729 | 0.633 | 0.87 |

MLA p50 breakdown:

- Metadata is flat at ~0.058 ms.
- Cached remap is flat at ~0.040 ms.
- Pack+all-gather grows with transferred bytes: 0.124 ms at 4k, 0.542 ms at 120k.
- Unpack grows with final dense bytes: 0.055 ms at 4k, 0.125 ms at 120k.

MLA conclusion:

- With final logical-dense unpack included, owner-packed all-gather is still not
  worth it at 4k.
- It starts becoming useful around 8k for transfer-only, and around 16k once
  remap is included.
- From 16k to 120k it is consistently about 13–19% faster by p50 in this
  microbenchmark.

Index p50 GPU ms from this run:

| tokens | dense all-reduce | gather+unpack | ratio |
| ---: | ---: | ---: | ---: |
| 4096 | 0.196 | 0.230 | 1.18 |
| 8192 | 0.197 | 0.233 | 1.18 |
| 16384 | 0.204 | 0.228 | 1.12 |
| 32768 | 0.210 | 0.136 | 0.65 |
| 65536 | 0.138 | 0.138 | 1.00 |
| 98304 | 0.154 | 0.140 | 0.91 |
| 122880 | 0.176 | 0.156 | 0.88 |

Index caveat:

- Index timings are small and showed more run-to-run inconsistency than MLA.
- Keep the earlier conclusion conservative: do not switch index to
  owner-packed all-gather based only on this microbenchmark until the production
  critical path and warmup/order effects are isolated.

### C90 — 2026-05-31 multi-extend prefixes break single-zigzag ordered all-gather assumption

Finding:

- The two-pass ordered all-gather idea only directly applies when the requested
  prefix is one contiguous in-seq CP split extent whose logical pages follow a
  single zigzag owner run:
  `0,1,...,cp_size-1,cp_size-1,...,1,0`.
- Production prefixes can be assembled from multiple historical extend/cache
  extents through radix/HiCache reuse. The global logical prefix can therefore
  look like a concatenation of several independently created zigzag runs, not one
  single zigzag over the full prefix length.
- A global two-pass all-gather keyed only by CP rank would mis-order pages for
  these multi-extent prefixes unless it also carries extent/run metadata and
  performs the ordered gather per run, or falls back to the generic rank-major
  gather plus logical-dense unpack.

Implication:

- Treat single-zigzag two-pass ordered all-gather as a fast path gated by an
  explicit single-run/regular-run descriptor check.
- Keep the generic rank-major + unpack path as the correctness baseline for
  multi-extend prefixes until segmented ordered-gather is implemented and
  benchmarked.

### C91 — 2026-05-31 custom P2P allgatherv candidate to replace dense all-reduce

Finding:

- Dense all-reduce is the wrong semantic primitive for CP shared KV materialize:
  each rank owns a sparse subset of pages, but all-reduce communicates the full
  dense buffer and performs a reduction even though non-owner entries are zeros.
- Padded owner-packed all-gather removes the reduction and most zero traffic, but
  still requires equal per-rank payload size (`max_owned_pages`) and an unpack to
  logical dense order.
- Production prefixes may be composed from multiple extend/cache runs, so a
  single ordered zigzag all-gather is only a gated fast path, not the generic
  replacement.

Candidate generic replacement:

- Implement a custom P2P ring allgatherv for owner page payloads:
  1. All ranks derive per-owner page counts and compact offsets from the same
     `logical_pages`/layout metadata; no runtime size collective is needed in the
     common deterministic case.
  2. Each rank packs only its owned pages into a compact contiguous segment.
  3. Ring P2P forwards variable-size segments for `cp_size - 1` steps using
     `batch_isend_irecv`/NCCL P2P.
  4. Receives write directly into a compact owner-concat buffer at known offsets.
  5. Existing logical-dense unpack can consume a compact page inverse
     (`logical_slot -> owner_offset + local_owner_ordinal`) instead of padded
     `rank * max_owned_pages + ordinal`.

Risks / benchmark requirements:

- Naive direct send-to-all is not acceptable; it increases per-rank injection
  versus ring all-gather. The P2P implementation must use a ring or another
  bandwidth-efficient schedule.
- P2P has more launch/control overhead than one NCCL collective. It must be
  benchmarked against both dense all-reduce and padded all-gather+unpack for
  4k--120k prefixes and both MLA/index payloads.
- NCCL P2P is not guaranteed to be zero-SM; do not claim compute-overlap benefits
  without profiling.

### C92 — 2026-05-31 CUDA IPC P2P ops are viable but should reuse existing SGLang IPC infrastructure

Finding:

- The repo already has CUDA IPC support in several places:
  - `python/sglang/srt/distributed/device_communicators/cuda_wrapper.py`
    exposes `cudaIpcGetMemHandle` / `cudaIpcOpenMemHandle`.
  - `python/sglang/srt/distributed/device_communicators/custom_all_reduce.py`
    and `custom_all_reduce_utils.py` already register/open IPC buffers and test
    real P2P access.
  - `python/sglang/jit_kernel/include/sgl_kernel/distributed/custom_all_reduce.cuh`
    already caches opened IPC handles and maintains peer pointer tables.
- Therefore a CP shared KV CUDA-IPC transport should not start from scratch. It
  should reuse the existing handle exchange, peer-access validation, and handle
  lifetime patterns, then add a narrower owner-page copy/gather primitive.

Design direction:

- Build an intra-node CUDA IPC owner-page transport for CP ranks on the same host.
- Register long-lived staging/output buffers once per scheduler/rank, exchange IPC
  handles during CP group initialization, and keep opened peer pointers cached.
- Launch a custom copy kernel that pulls remote owner page segments from peer
  pointers into the local compact owner-concat or logical-dense destination.
- Keep NCCL/PyTorch collective fallback only for unsupported topology or failed
  IPC/P2P validation, and log it explicitly.

Constraints:

- CUDA IPC is intra-node only. It does not replace RDMA or cross-node transfer.
- Correctness requires explicit stream/event ordering: a rank must not read a
  peer staging segment before the peer pack kernel has completed.
- Lifetime must be explicit: registered peer buffers cannot be freed/reallocated
  while another rank holds an IPC pointer.
- This path may still use SMs for the copy kernel unless implemented with copy
  engine primitives; do not assume it is 0-SM without profiling.

### C93 — 2026-05-31 tai-kernel is the right boundary for CUDA IPC owner-page gather

Finding:

- `tai-kernel` already has a JIT C++/CUDA extension path for NSA prefill KV IO:
  `python/tai_kernel/nsa_prefill/_kvcacheio_extension_loader.py` builds
  `csrc/kvcacheio_ops.cc` and `csrc/kvcacheio_lf_pf.cu` with
  `torch.utils.cpp_extension.load` and registers `tai_kernel_ops` via
  `TORCH_LIBRARY_FRAGMENT`.
- CUDA IPC cannot be implemented in Triton alone because handle creation/opening,
  peer pointer lifetime, and optional interprocess event handling require CUDA
  runtime APIs. The right shape is a small C++/CUDA extension in `tai-kernel`
  plus CUDA kernels for peer-pointer copy/gather.

Design implication:

- Add IPC primitives to the existing tai-kernel extension boundary rather than
  adding another SGLang-local extension or relying on Python/ctypes in the hot
  path.
- Expose handles as fixed-size uint8 tensors or keep opened peer pointers in a
  C++ registry keyed by an integer transport id. Avoid Python object handles in
  production hot paths.

### C94 — 2026-05-31 CUDA IPC handle offset issue with PyTorch allocator tensors

Finding:

- A first distributed CUDA IPC logical-dense gather using PyTorch-allocated
  staging tensors produced correct self-owned pages but corrupted remote-owned
  pages. The failure pattern was roughly `1 / cp_size` correct data and the rest
  invalid, which pointed to remote IPC pointer interpretation rather than owner
  metadata.
- The root cause is that CUDA IPC exports an allocation handle, while PyTorch's
  caching allocator may place a tensor data pointer at an internal offset within
  a larger allocation. Opening the IPC handle on a peer can return the allocation
  base, not the tensor's exact `data_ptr()`. Existing SGLang custom all-reduce
  code avoids this by using long-lived `cudaMalloc` IPC buffers or by carrying
  `(handle, offset)` metadata.

Resolution in the prototype:

- `tai-kernel` now has a CUDA IPC-exportable staging allocation helper backed by
  `cudaMalloc`. The benchmark copies compact owner pages into this pointer-stable
  byte buffer before exporting the handle.
- Production integration must either use long-lived `cudaMalloc` staging buffers
  or carry and apply explicit IPC handle offsets. Do not export arbitrary
  PyTorch cache-allocator tensors and assume the opened peer pointer equals
  `data_ptr()`.

### C95 — 2026-05-31 CUDA IPC P2P owner-page benchmark evidence

Remote benchmark log:

- `/mnt/beegfs/cjy/log/cp_shared_kv_ipc_gather_segments_20260531_092319.log`

Configuration:

- 8 local CP ranks on g0034, BF16 MLA payload, `page_size=64`, `kv_dim=576`.
- Prefix tokens: `4096 8192 16384 32768 65536 98304 122880`.
- Paths:
  - `dense_all_reduce`: local dense clone + NCCL all-reduce.
  - `cuda_ipc_logical_dense_gather`: one SM kernel directly loads peer pages into
    logical-dense order.
  - `cuda_ipc_owner_concat_segments`: one `cudaMemcpyAsync` segment copy per peer
    into owner-concat layout; not final consumer layout.
  - `cuda_ipc_segments_reorder_logical`: segment copy plus logical-dense unpack.

p50 GPU ms:

| tokens | dense all-reduce | IPC direct logical | IPC owner-concat | IPC segment+unpack |
| ---: | ---: | ---: | ---: | ---: |
| 4096 | 0.091 | 0.092 | 0.065 | 0.100 |
| 8192 | 0.115 | 0.160 | 0.093 | 0.108 |
| 16384 | 0.156 | 0.297 | 0.189 | 0.198 |
| 32768 | 0.234 | 0.573 | 0.337 | 0.371 |
| 65536 | 0.392 | 1.177 | 0.662 | 0.725 |
| 98304 | 0.553 | 1.828 | 0.983 | 1.091 |
| 122880 | 0.666 | 2.316 | 1.261 | 1.347 |

Conclusion:

- The naive CUDA IPC direct logical-dense kernel is not a production replacement
  for NCCL all-reduce. It uses SM peer loads and becomes much slower as prefix
  grows.
- Segment-level `cudaMemcpyAsync` P2P is better than direct page-gather but is
  still slower than NCCL dense all-reduce for large MLA prefixes once the final
  logical-dense layout is required.
- The useful takeaway is architectural, not performance-ready: CUDA IPC is viable
  as a transport substrate, but a production win likely requires a more fused
  layout contract that avoids a full logical-dense rewrite or uses a consumer
  that can read owner-concat/segmented layout directly. Do not replace the current
  materialize collective with this prototype path.

### C96 — 2026-05-31 attention idx/page correspondence is the real owner-concat contract

Finding:

- The attention-side requirement is not that the materialized KV buffer must be
  physically sorted by logical page. The hard requirement is that every attention
  idx/loc points to the page slot that contains the corresponding token/page.
- Current CP shared KV already has this abstraction through `page_inverse` and
  `remap_logical_locs_to_slot_dense_locs`: logical token locs are rewritten to
  dense slot locs before consumers read the materialized buffer.

Implication:

- An owner-concat or segmented buffer can be a valid consumer layout if we build
  `page_inverse[logical_page] = owner_concat_slot` and remap attention locs to
  `owner_concat_slot * page_size + offset`.
- This avoids requiring a full owner-concat -> logical-dense rewrite, as long as
  every downstream consumer uses remapped locs/page ids and no kernel assumes
  `physical_slot == logical_page` or monotonic logical ordering.
- The production audit should therefore focus on consumers that bypass the
  `page_inverse` remap or derive page slots directly from logical positions.

### C97 — 2026-05-31 CUDA IPC fused materialize must live in CUDA extension, not Triton

Finding:

- Triton kernels can remap/materialize a local tensor that already exists in the
  current process, but they cannot own CUDA IPC handle creation/opening or safely
  dereference peer IPC pointer tables by themselves.
- The first slot-dense materialize helper was useful for local owner-concat
  tensors, but it is not the correct primitive for a fused CUDA IPC transport.
  The fused IPC path belongs in `tai-kernel`'s C++/CUDA extension.

Implemented prototype:

- Added a tai-kernel CUDA extension op:
  `materialize_cuda_ipc_peer_pages_slot_dense(peer_ptrs, dst, owner_ranks,
  src_page_indices, page_nbytes, ...)`.
- The op writes the current SGLang consumer contract directly:
  - dense page 0 is zeroed as dummy,
  - valid slot `i` copies
    `peer_ptrs[owner_ranks[i]][src_page_indices[i]] -> dst[i + 1]`,
  - negative owner/source entries are zero-filled sentinel slots.
- This keeps attention/index consumers on the existing slot-dense contract while
  allowing the transport side to experiment with owner-packed IPC staging.

Remote verification:

- Unit:
  `PYTHONPATH=python python -m pytest -q tests/nsa_prefill/test_cuda_ipc_gather.py`
  on `g0034` passed: `4 passed`.
- Benchmark log:
  `/mnt/beegfs/cjy/log/cp_shared_kv_ipc_fused_materialize_20260531_175520.log`.

Benchmark result summary, BF16 MLA page payload, 8 CP ranks:

| tokens | dense all-reduce | IPC segment+slot-dense | IPC direct peer slot-dense |
| ---: | ---: | ---: | ---: |
| 4096 | 0.100 ms | 0.094 ms | 0.092 ms |
| 8192 | 0.122 ms | 0.108 ms | 0.160 ms |
| 16384 | 0.158 ms | 0.207 ms | 0.297 ms |
| 32768 | 0.234 ms | 0.402 ms | 0.578 ms |
| 65536 | 0.390 ms | 0.752 ms | 1.196 ms |
| 98304 | 0.551 ms | 1.143 ms | 1.812 ms |
| 122880 | 0.668 ms | 1.366 ms | 2.319 ms |

Conclusion:

- The direct SM CUDA IPC peer-load fused materialize is correctness-useful but
  was not a production replacement for NCCL all-reduce on large prefixes under
  the initial under-parallel launch configuration.
- The segment-copy + local materialize path is better than per-page peer loads,
  but still loses to dense all-reduce for large MLA prefixes in this benchmark.
- Do not wire this prototype into production hot paths as a default. The next
  performance direction should be either a copy-engine/contiguous segment
  transport that avoids SM peer loads, or a consumer contract that can consume
  owner-concat/segmented layout without a full rewrite.

### C98 — 2026-05-31 SM fused IPC materialize launch tuning changes the conclusion

Correction:

- C97's negative performance conclusion was caused by a bad launch policy, not
  by the fused materialize mapping itself. The default wrapper used
  `block_quota=2, num_warps_per_block=32`, which effectively capped the copy
  kernel around two CTAs. For a 72 KiB BF16 MLA page, each warp then copied many
  pages serially and underutilized the GPU badly.
- The current short-term goal is explicitly SM-consuming low latency/high
  throughput. Low-SM/copy-engine prefetch friendliness is deferred.

Change:

- The CUDA IPC gather/materialize Python wrappers and benchmark now default to
  `block_quota=256, num_warps_per_block=8`.
- This keeps enough CTAs active for large prefixes while avoiding 1024-thread
  CTAs from the old `32`-warp default.

Remote benchmark log:

- `/mnt/beegfs/cjy/log/cp_shared_kv_ipc_sm_tuned_20260531_181926.log`

Configuration:

- 8 local CP ranks on g0034, BF16 MLA payload, `page_size=64`, `kv_dim=576`.
- Prefix tokens: `4096 8192 16384 32768 65536 98304 122880`.
- `block_quota=256`, `num_warps_per_block=8`.

p50 GPU ms:

| tokens | dense all-reduce | IPC direct logical | IPC direct slot-dense |
| ---: | ---: | ---: | ---: |
| 4096 | 0.100 | 0.090 | 0.090 |
| 8192 | 0.118 | 0.091 | 0.091 |
| 16384 | 0.160 | 0.096 | 0.096 |
| 32768 | 0.236 | 0.138 | 0.138 |
| 65536 | 0.393 | 0.237 | 0.240 |
| 98304 | 0.554 | 0.347 | 0.347 |
| 122880 | 0.668 | 0.421 | 0.421 |

Conclusion:

- Under the tuned SM launch, direct IPC peer-page fused materialize beats dense
  all-reduce across the tested 4k--120k MLA prefix range.
- The slot-dense variant has essentially the same cost as logical-dense gather,
  so preserving the existing SGLang consumer contract is not the dominant cost.
- The segment-copy path is now slower than tuned direct peer-page gather for this
  benchmark. Do not prefer owner-concat segment copy solely for performance
  unless a future consumer can avoid the final rewrite entirely.

### C99 — 2026-05-31 multi-warp/page IPC materialize tuning

Change:

- Added a benchmark-selectable `warps_per_page` mode to tai-kernel
  `materialize_cuda_ipc_peer_pages_slot_dense`.
- `warps_per_page=1` preserves the previous one-warp-per-page mapping.
- `warps_per_page>1` assigns a whole warp group to one page and stripes the
  page copy across `warps_per_page * 32` threads.
- Python wrapper default is now `warps_per_page=0`, an auto selector based on
  the measured H200 regions for the production ~72 KiB BF16 MLA page:
  - <= 256 pages: 8 warps/page,
  - 512--1024 pages: 1 warp/page,
  - >= 1536 pages: 4 warps/page,
  - synthetic pages smaller than 64 KiB: 1 warp/page.

Remote correctness:

- `PYTHONPATH=python python -m pytest -q tests/nsa_prefill/test_cuda_ipc_gather.py`
  on `g0034` passed: `6 passed`.

Remote benchmark logs:

- Variant sweep:
  `/mnt/beegfs/cjy/log/cp_shared_kv_ipc_warps_per_page_20260531_114147.log`
- Auto selector sweep:
  `/mnt/beegfs/cjy/log/cp_shared_kv_ipc_auto_wpp_20260531_114447.log`

Auto selector p50 GPU ms, BF16 MLA page payload, 8 CP ranks:

| tokens | pages | dense all-reduce | IPC direct slot-dense auto | speedup vs dense |
| ---: | ---: | ---: | ---: | ---: |
| 4096 | 64 | 0.093 | 0.057 | 1.63x |
| 8192 | 128 | 0.115 | 0.060 | 1.92x |
| 16384 | 256 | 0.164 | 0.083 | 1.98x |
| 32768 | 512 | 0.235 | 0.134 | 1.75x |
| 65536 | 1024 | 0.394 | 0.236 | 1.67x |
| 98304 | 1536 | 0.554 | 0.348 | 1.59x |
| 122880 | 1920 | 0.670 | 0.421 | 1.59x |

Bandwidth reading for the largest case:

- `122880` tokens = `1920` pages = `135 MiB` logical payload per GPU.
- `0.421 ms` gives about `336 GB/s` payload rate per GPU.
- The kernel performs roughly one full payload read plus one full payload write,
  so effective HBM traffic is about `672 GB/s`, ~14% of H200's ~4.8 TB/s HBM
  bandwidth.
- Remote NVLink receive traffic is about `7/8` of the payload, or `294 GB/s`,
  ~61% of the per-direction 18-link H200 NVLink budget (`18 * 26.562 GB/s ~= 478 GB/s`).

Conclusion:

- This kernel is no longer obviously HBM-limited; NVLink plus per-page peer-load
  overhead is closer to the limiting side for large prefixes.
- The remaining large-prefix headroom to the per-direction NVLink roof is about
  `478 / 294 ~= 1.6x` under this access pattern.
- For small prefixes, the main gain came from increasing per-page parallelism
  rather than more CTAs; launch/fixed overhead still dominates below ~8k tokens.

Additional C99 correctness check:

- Ran the multi-rank benchmark with checks enabled for `4096` and `122880`
  tokens and `warps_per_page=0`; all benchmark reference assertions passed.

### C100 — 2026-05-31 vectorized 16B copy for multi-warp/page IPC materialize

Change:

- The multi-warp/page IPC materialize path now uses `ld.global.nc.v2.u64` +
  `st.global.cg.v2.u64` when the page payload is 16-byte aligned.
- The one-warp/page legacy path is unchanged; the vectorized helper only affects
  the measured multi-warp variants used by the auto selector.

Remote verification:

- `PYTHONPATH=python python -m pytest -q tests/nsa_prefill/test_cuda_ipc_gather.py`
  on `g0034` passed: `6 passed`.
- Multi-rank benchmark checks enabled for `4096` and `122880` tokens passed.

Remote benchmark log:

- `/mnt/beegfs/cjy/log/cp_shared_kv_ipc_auto_wpp_vec16_20260531_114903.log`

Auto selector p50 GPU ms after vectorization:

| tokens | pages | dense all-reduce | IPC direct slot-dense auto | previous auto IPC | delta |
| ---: | ---: | ---: | ---: | ---: | ---: |
| 4096 | 64 | 0.137 | 0.042 | 0.057 | -26% |
| 8192 | 128 | 0.112 | 0.052 | 0.060 | -13% |
| 16384 | 256 | 0.162 | 0.082 | 0.083 | -1% |
| 32768 | 512 | 0.236 | 0.136 | 0.134 | +1% |
| 65536 | 1024 | 0.392 | 0.238 | 0.236 | +1% |
| 98304 | 1536 | 0.553 | 0.339 | 0.348 | -3% |
| 122880 | 1920 | 0.668 | 0.416 | 0.421 | -1% |

Bandwidth reading for `122880` tokens:

- Payload rate: `135 MiB / 0.416 ms ~= 340 GB/s` per GPU.
- Effective HBM read+write traffic: `~680 GB/s`, still only ~14% of H200 HBM.
- Remote NVLink receive traffic: `(7/8) * 340 ~= 298 GB/s`, ~62% of the
  `~478 GB/s` per-direction H200 NVLink link budget.

Conclusion:

- 16B vectorization materially improves the short-prefix auto path and slightly
  improves the large-prefix path; it does not change the bottleneck conclusion.
- Large-prefix headroom remains primarily in reducing peer-load overhead / better
  NVLink utilization, not in HBM bandwidth.

### C101 — 2026-05-31 index page payload also benefits from IPC materialize

Clarification:

- The descriptor builder needed by SGLang runtime is the translation from a
  consumer slot page table to the IPC kernel descriptors:
  `slot_logical_pages -> owner_ranks, src_page_indices`.
- The IPC kernel itself does not understand radix/HiCache/current-reuse state;
  it only consumes owner/source descriptors. Full hit, multi-extend hit, and
  partial-current prefix reuse are all covered if the runtime builds these
  descriptors from the same page-aligned slot contract.

Index payload benchmark:

- NSA index page buffer is `uint8` with per-token bytes
  `index_head_dim + index_head_dim / quant_block_size * 4 = 128 + 4 = 132`.
- With `page_size=64`, each index page is `8448` bytes.
- Benchmark command used `--dtype uint8 --kv-dim 132` to match the real index
  page payload size and dtype.

Remote log:

- `/mnt/beegfs/cjy/log/cp_shared_kv_ipc_index_uint8_20260531_120416.log`

p50 GPU ms, 8 CP ranks:

| tokens | pages | dense uint8 all-reduce | IPC direct slot-dense | speedup |
| ---: | ---: | ---: | ---: | ---: |
| 4096 | 64 | 0.119 | 0.033 | 3.61x |
| 8192 | 128 | 0.070 | 0.031 | 2.26x |
| 16384 | 256 | 0.070 | 0.033 | 2.12x |
| 32768 | 512 | 0.081 | 0.037 | 2.19x |
| 65536 | 1024 | 0.102 | 0.049 | 2.08x |
| 98304 | 1536 | 0.120 | 0.061 | 1.97x |
| 122880 | 1920 | 0.139 | 0.072 | 1.93x |

Additional check:

- Multi-rank correctness benchmark with checks enabled for `4096` and `122880`
  index payloads passed.

Conclusion:

- IPC materialize is beneficial for index pages too. Because index pages are
  much smaller than MLA pages, the absolute saving is smaller, but the relative
  saving versus uint8 all-reduce is still about `~2x` for large prefixes and
  stronger for short prefixes.
- This supports wiring both MLA and index full/partial prefix materialize to the
  same descriptor-driven IPC path.

### C102 — 2026-05-31 runtime IPC materialize is wired only for synchronous non-prefetch paths

Implementation note:

- The first SGLang runtime hookup targets only the synchronous materialize paths:
  - MLA full materialize with a precomputed slot remap,
  - MLA partial-current synchronous prefix materialize,
  - index full materialize with a precomputed slot remap,
  - index partial-current synchronous prefix materialize.
- `CpSharedKVMlaPrefetcher` and `CpSharedKVIndexPrefetcher` are intentionally
  unchanged in this slice.  Async prefetch still uses the existing local
  materialize + async collective path until stream-ordering and CPU-submit cost
  are re-evaluated separately.
- The runtime descriptor contract is:
  `slot_logical_pages -> owner_ranks, src_page_indices`, where source page index
  is the peer CP-local physical page id including dummy page 0.  Invalid slots
  use `-1/-1` and are zero-filled by the tai-kernel IPC slot-dense materialize
  kernel.
- CUDA IPC handle exchange is cached per long-lived source tensor pointer/shape
  and CP group.  The cache intentionally avoids per-layer handle collectives;
  a tiny handle/offset all-gather is still required the first time a KV/index
  source buffer is registered on a rank.

Important correction:

- Runtime IPC must carry CUDA allocation offsets.  Production source tensors are
  PyTorch caching-allocator tensors, not guaranteed `cudaMalloc` base pointers.
  `tai-kernel` now exposes `get_cuda_ipc_mem_handle_with_offset()` and
  `open_cuda_ipc_mem_handles_with_offsets()` so peers dereference the exact
  tensor data pointer instead of the allocation base.

Risk / not yet covered:

- Peer pointers are cached for long-lived KV/index buffers and are not currently
  closed before process exit.  This is acceptable for the scheduler lifetime but
  should not be reused for short-lived staging tensors.
- The IPC path currently falls back to the old collective materialize path if the
  tai-kernel IPC capability is missing or peer opening fails; the fallback logs
  `[CP_SHARED_KV_FALLBACK][tai_ipc_materialize]`.  Once remote ETE validates the
  capability, this can be tightened to fail-fast for the production GLM5 config.
- Remote CUDA verification is still required after syncing both `tai-kernel` and
  `sglang-dev`; local CPU import is blocked by missing optional dependencies.

### C103 — 2026-06-01 FP8 MLA current reuse must splice packed cache rows, not bf16 compact rows

Problem found:

- NSA FP8 KV cache stores MLA rows in the packed persistent layout:
  `[512 FP8 nope bytes][4 FP32 scales][64 BF16 rope values]`, i.e. `656` bytes
  per token row.
- The CP shared-KV partial-current MLA path previously built fresh current rows
  with `cat([k_nope, k_rope])`, i.e. the direct bf16/fp16 `576`-wide layout.
  That is only valid for non-FP8 cache.  In FP8 mode it would splice rows whose
  dtype/row-width do not match the materialized prefix buffer.

Implementation direction:

- `tai-kernel` now provides `pack_quant_mla_kv(k_nope, k_rope) -> uint8[N,1,656]`
  by reusing the fused quantized MLA store kernel with identity logical locs.
  This is correctness-first and keeps the quantization/store contract in TAI
  rather than rebuilding it with torch ops in SGLang.
- SGLang detects packed FP8 MLA KV cache by row width `656` plus `uint8` or
  `float8_e4m3*` dtype, then packs current rows before current-only or
  partial-current reuse.
- `fp8_e5m2` remains fail-fast for NSA CP shared-KV FP8 reuse; current model
  config only supports `fp8_e4m3` for NSA.
- No silent bf16 fallback is allowed for FP8 current reuse.  Missing TAI pack
  capability raises `[CP_SHARED_KV_FAIL_FAST][fp8_current_pack]`.

IPC coverage:

- The descriptor-driven IPC materialize kernel is byte-count based.  It already
  covers FP8 MLA pages when `page_nbytes = page_size * 656`, so no separate IPC
  copy kernel is required for packed FP8 rows.
- A CUDA test was added to lock the FP8 packed MLA page size through the IPC
  slot-dense materialize path.

Verification status:

- Local and remote `py_compile` passed for touched SGLang and TAI files.
- Local source-level backend hook test passed.  Local dynamic helper tests remain
  blocked by missing local `orjson`; the remote container is authoritative for
  those tests.
- Remote `g0034` SGLang targeted tests passed for FP8 current-pack success,
  fail-fast-on-missing-TAI, and backend current-reuse hook detection.
- Remote `g0034` TAI CUDA tests passed for `pack_quant_mla_kv` identity layout
  and FP8 packed MLA page IPC slot-dense materialize.

### C104 — 2026-06-01 current-only MLA reuse must still return page-slot KV buffers

Problem found:

- FP8 current-only CP shared-KV reuse packed the freshly computed current rows
  into a compact tensor shaped like `[valid_tokens, 1, 656]` and assigned it
  directly to `kv_cache`.
- `flashmla_kv` consumes MLA KV cache as pages and reshapes it with
  `view(-1, page_size, 1, kv_cache_dim)`.  A short current-only request such as
  4 tokens therefore produced `4 * 656 = 2624` elements, which cannot be viewed
  as full 64-token pages.
- This violated the page-aligned cache contract.  FP8 does not get an exception:
  even packed FP8 MLA rows must be stored and exposed through page-sized slots.

Implementation direction:

- The current-only branch now uses the same page-slot compose helper as
  partial-current reuse, with `prefix_pages=0`.
- `prefix_pages=0` means there are no historical prefix pages to materialize or
  reduce.  It does **not** mean the output buffer has zero pages.  The current
  suffix still occupies `ceil(valid_tokens / page_size)` physical page slots.
- The compact current tensor remains only an input to the slot-fill step.  The
  attention-facing `kv_cache` is the composed page buffer, and returned locs are
  remapped into that page buffer with non-current tail slack masked out.

Verification status:

- Local source-level regression test passed for the backend current-only branch:
  it now requires `materialize_prefix_and_reuse_current_kv_page_slots`,
  `prefix_pages=0`, and forbids `kv_cache = current_kv_cache`.
- Local `py_compile` passed for `nsa_backend.py` and `cp_shared_kv_runtime.py`.
- Remote `g0034` container `py_compile` passed for `nsa_backend.py` and
  `cp_shared_kv_runtime.py`.
- Remote `g0034` targeted runtime tests passed for current-only page-slot
  layout, padded-current reuse gating, FP8 current-pack success, and
  fail-fast-on-missing-TAI.
- Local dynamic runtime helper tests are still blocked by missing optional local
  dependency `orjson`; the remote container remains authoritative for those
  tests.

### C105 — 2026-06-01 current-only tiny suffixes must also skip NSA in-seq CP split

Problem found:

- The short-page CP-split guard only handled radix-hit suffixes with a positive
  page-aligned prefix.  Current-only FP8 warmup/first-token paths with
  `prefix_len=0` and `extend_len=64` still entered NSA in-seq CP split.
- With `cp_size=8` and `page_size=64`, that is only one physical page.  The
  page-aligned in-seq split produces mostly zero-token CP lanes.  The downstream
  FlashMLA KV path then sees inconsistent q-batch / metadata cardinality and the
  kernel rejects `num_splits` with `num_splits must have shape (b+1)`.

Implementation direction:

- The page-unit guard now treats `prefix_len=0` as a valid page-aligned prefix
  for the purpose of deciding whether CP split is safe.
- If `ceil(extend_len / page_size) < cp_size`, CP split is skipped for CP shared
  KV regardless of whether the request is current-only or a radix hit.
- This preserves the page cache contract: we avoid splitting a single page over
  eight owner lanes instead of falling back to token-balanced sub-page splits.

Verification status:

- Added a unit regression that `extend_len=64`, `prefix_len=0`, `page_size=64`,
  `cp_size=8` does not use NSA in-seq CP split.
- Remote `g0034` container `py_compile` passed for `nsa/utils.py`.
- Remote `g0034` targeted tests passed for the new current-only skip guard and
  adjacent short-page / one-page-per-rank CP-split behavior.
- Remote `g0034` combined targeted tests passed for the current-only page-slot
  FP8 reuse checks plus the new CP-split guard.

### C106 — 2026-06-01 effective NSA impl must drive metadata and top-k layout

Finding:

- `forward_extend()` selects the effective NSA implementation from forward mode:
  target-verify and draft-extend use `nsa_decode_impl`, while normal extend uses
  `nsa_prefill_impl`.
- Two metadata/layout decisions still used global fields instead of that effective
  implementation:
  - `flashmla_metadata` construction in normal `init_forward_metadata()` was gated
    only by `nsa_decode_impl == "flashmla_kv"`.
  - `get_topk_transform_method()` selected ragged-vs-paged from
    `nsa_prefill_impl` only.

Impact:

- The default Hopper FP8 setup (`prefill=flashmla_auto -> flashmla_kv`,
  `decode=flashmla_kv`) did not expose this, but explicit backend combinations
  could mismatch: e.g. decode `flashmla_kv` with prefill `flashmla_sparse` could
  make target/draft use FlashMLA-KV attention with ragged top-k metadata.
- This is the same class of inconsistency as the recent FP8 failures: attention
  kernel selection, top-k/index layout, and FlashMLA scheduler metadata must be
  derived from one effective implementation for the current forward mode.

Correction:

- Added `_effective_nsa_impl_for_forward_batch()` and routed normal metadata
  creation, forward-layer top-k transform selection, and indexer metadata through
  it.
- CUDA-graph decode metadata remains decode-backend gated because that path is
  decode/verify/draft graph state, not normal prefill metadata.

Verification:

- Remote g0034 container `py_compile` passed for `nsa_backend.py` and the runtime
  test file.
- Remote targeted tests passed for effective top-k transform, FlashMLA metadata
  source gating, and current-only page-slot reuse.

### C107 — 2026-06-01 non-page-aligned CP shared-KV prefixes must fail before CP split

Finding:

- Production radix/HiCache prefix matching is supposed to floor CP shared-KV
  cache hits to page boundaries before scheduling. Evidence:
  - base radix match floors keys to page size;
  - HiRadix CP paths floor exact valid-tail hits and backed partial-tail splits;
  - scheduler runtime checks require `cache_protected_len % page_size == 0`.
- The CP split builder still has a legacy token-balanced fallback when called
  with a non-page-aligned `extend_prefix_len`. If production ever reaches that
  branch, CP split would reintroduce sub-page split semantics instead of exposing
  the upstream page-flooring bug.

Correction:

- `can_cp_split()` now fails fast for single-request CP shared-KV in-seq split
  when `extend_prefix_lens_cpu[0]` is not page-aligned.
- This does not change the normal current-only path (`prefix_len=0`) or valid
  page-aligned radix-hit path. It only makes an invalid production state explicit.

Verification status:

- Added unit coverage that a non-page-aligned CP shared-KV prefix raises
  `[CP_SHARED_KV_FAIL_FAST][cp_split_non_page_aligned_prefix]` before CP split.
- Remote g0034 targeted test passed for non-page-aligned prefix fail-fast plus adjacent current-only and page-aligned CP-split cases.

### C108 — 2026-06-01 current-only index reuse tests must model the real CPU-length contract

Finding:

- A wider remote unit run exposed a stale test stub for NSA index current-only
  reuse. The production gate is `is_current_only_extend_batch()`, which requires
  `forward_mode.is_extend_without_speculative()`, zero prefix lengths, and
  `seq_lens_cpu == extend_seq_lens_cpu`.
- The test passed `current_index_kv` but omitted those CPU metadata fields, so
  the production helper correctly treated it as not current-only and attempted
  shared index materialization.

Correction:

- The test stub now carries the same CPU-length contract as production current-only
  batches. This keeps the assertion meaningful: current-only index reuse skips
  shared index materialization only when the batch metadata proves that all
  visible rows are current rows.

Verification:

- Remote g0034 container two-file unit suite passed:
  `test_nsa_cp_utils.py` + `test_cp_shared_kv_runtime.py`
  (`124 passed, 5 warnings, 2 subtests passed`).
- Remote g0034 container page-aligned/cache runtime suite passed:
  `test_nsa_cp_utils.py` + `test_cp_shared_kv_layout.py` +
  `test_cp_shared_kv_runtime.py`
  (`149 passed, 5 warnings, 2 subtests passed`).

### C109 — 2026-06-01 FP8 HiCache target/draft budget split uses packed row bytes

Finding:

- CP HiCache shares `--hicache-size` across target and EAGLE/NextN draft when a
  draft KV pool is attached.
- The split is byte-based but chooses a target token capacity first:
  `target_pages = budget / ((target_bytes_per_token + draft_bytes_per_token) * page_size)`.
  The remaining bytes go to draft, and draft capacity must be at least target
  capacity.
- For NSA FP8, `NSATokenToKVPool.kv_cache_dim` is the packed persistent row width
  (`656` uint8 bytes/token/layer), not the BF16 compact MLA width.  HiCache host
  pool construction passes the device `kv_cache_dim` as `override_kv_cache_dim`,
  so `NSATokenToKVPoolHost.get_size_per_token()` and
  `_estimate_hicache_size_per_token()` agree.
- GLM-5 EAGLE draft has one executable draft layer.  Therefore under FP8 the
  draft host allocation can be small in bytes (for example around 1/78 of the
  target KV bytes) while still having equal or slightly larger token capacity.
  A small draft GB log is not by itself a capacity bug.

Correction:

- No production code change was needed for the current target/draft HiCache size
  split.  Added regression tests to pin the FP8 packed-row byte contract and the
  one-layer draft capacity implication.

Verification status:

- Remote g0034 container targeted FP8 HiCache budget tests passed
  (`4 passed, 3 warnings`).
- Remote g0034 container full HiCache metadata unit file passed
  (`102 passed, 5 warnings`).

### C110 — 2026-06-01 FP8 FlashMLA-KV prefill is incompatible with NSA in-seq CP split metadata

Finding:

- Remote g0034 prefill log `sglang_cp_hicache_20260531_183930.log` died on all
  CP ranks with `RuntimeError: num_splits must have shape (b+1)` inside
  `flash_mla_with_kvcache()`.
- Startup was Hopper FP8 KV cache with `nsa_prefill_backend=flashmla_auto`,
  `nsa_decode_backend=flashmla_kv`, `--enable-nsa-prefill-context-parallel`, and
  `--nsa-prefill-cp-mode in-seq-split`.
- The failing request was a normal target prefill/extend with about 40k input
  tokens and no cache prefix.  Auto dispatch selected `flashmla_kv` for FP8
  prefill on Hopper.
- In in-seq CP split, the model input hidden states are split by
  `prepare_input_dp_with_cp_dsa()`/`cp_split_and_rebuild_data()`, but normal NSA
  metadata still builds FlashMLA-KV scheduler metadata from the unsplit full
  `seqlens_expanded`.  `_forward_flashmla_kv()` then reshapes local CP q rows as
  `q=[local_q, 1, heads, dim]` while passing `num_splits` sized for the full q
  row count, violating the FlashMLA-KV `(b+1)` contract.
- BF16 did not hit this because the default prefill implementation is
  `flashmla_sparse`, which does not use FlashMLA-KV `num_splits` metadata.

Correction direction:

- Treat FP8 FlashMLA-KV as decode/verify/draft-only for NSA prefill CP until the
  in-seq CP metadata is made FlashMLA-KV-aware.
- Auto FP8 prefill under NSA CP should dispatch to `flashmla_sparse` instead of
  `flashmla_kv`; explicit invalid `flashmla_kv` prefill under active in-seq CP
  should fail loudly rather than reaching the kernel shape error.

Verification target:

- Add unit coverage for FP8 `flashmla_auto` + NSA prefill CP selecting
  `flashmla_sparse`.
- Sync to g0034 and rerun the remote unit slice before ETE restart.

C110 implementation note:

- `flashmla_auto` now selects `flashmla_sparse` for FP8 normal prefill whenever
  NSA prefill CP is enabled and the forward mode is context-parallel extend.
  This avoids constructing full-q FlashMLA-KV scheduler metadata before the
  model's in-seq CP split narrows q rows.
- If an explicit configuration still routes an actual CP-split FP8 prefill layer
  into `flashmla_kv`, `forward_extend()` raises
  `[NSA_FP8_FLASHMLA_KV_CP_PREFILL_UNSUPPORTED]` before calling the kernel.
  This is intentional: the alternative is the opaque kernel error
  `num_splits must have shape (b+1)`.

C110 verification update:

- Remote g0034 container `py_compile` passed for `nsa_backend.py` and
  `test_cp_shared_kv_runtime.py`.
- Remote targeted tests passed for FP8 auto prefill CP dispatch, explicit
  FlashMLA-KV CP prefill fail-fast, effective top-k transform selection, and
  FlashMLA metadata gating (`4 passed, 5 warnings`).
- Remote related unit sweep passed:
  `test_nsa_cp_utils.py`, `test_cp_shared_kv_layout.py`,
  `test_cp_shared_kv_runtime.py`, and `test_cp_hicache_metadata.py`
  (`253 passed, 5 warnings, 2 subtests passed`).

### C111 — 2026-06-01 TAI page_first_direct direct transfers must not hide index D2H copies

Finding:

- The TAI `cudaMemcpyBatchAsync` page_first_direct per-layer D2H/H2D paths were
  accepting `src_indices` and `dst_indices` tensors and then doing
  `src_indices.to(CPU).contiguous()` / `dst_indices.to(CPU).contiguous()` inside
  the C++ wrapper before building the memcpy descriptor arrays.
- If a benchmark or future caller passed CUDA indices, the direct path silently
  inserted a CUDA-to-host metadata copy and potential host synchronization before
  each transfer submission.  This made direct-path timing ambiguous and could
  hide CPU/control-path regressions.
- SGLang production `CacheController.move_indices()` already uses CPU indices
  for `io_backend=direct` + `page_first_direct`; the hidden conversion mainly
  affected tai-kernel tests/benchmarks and unsafe future callers.

Correction:

- Tighten the TAI per-layer direct D2H/H2D contract: `src_indices` and
  `dst_indices` must be CPU int64 contiguous tensors.  CUDA indices now fail
  fast with `[CP_HICACHE_FAILFAST][tai_page_first_direct_cpu_indices]` instead
  of being copied back to host.
- Update tai-kernel direct tests and benchmarks to pass CPU indices.  The
  `--indices-on-cuda` benchmark option is kept only as a deprecated error path.

Verification target:

- Build/reinstall tai-kernel remotely, then run the CUDA `test_kvcacheio_lf_pf.py`
  direct-path tests on g0034.  Local CUDA execution remains intentionally
  unsupported.

C111 verification update:

- Local tai-kernel `git diff --check`, Python `py_compile`, and CUDA-skipped
  local pytest collection passed.
- Synced tai-kernel direct-path code/tests/benchmarks to g0034 and ran remote
  container CUDA tests:
  - fail-fast CPU-index contract tests: `4 passed`
  - direct LF->PF/PF->LF reference tests: `8 passed`
  - full `tests/nsa_prefill/test_kvcacheio_lf_pf.py`: `23 passed`

### C112 — 2026-06-01 Benchmark GFD-style H2D alternatives against current page_first_direct baseline

Finding:

- The GFD article's diagnosis applies to many-small-copy H2D workloads, but the
  direct optimization must be compared against our current baseline, not against
  naive per-token `cudaMemcpyAsync`.
- While preparing that benchmark, an important layout constraint was identified:
  current HiCache `page_first_direct` host layout is `[page, layer, page_size,
  ...]`.  For a fixed `layer_id`, page `N` and page `N+1` are not adjacent in
  host memory; they are separated by the payloads of the other layers.  Therefore
  direct page-run coalescing for the existing per-layer PF->LF H2D path is not
  physically valid when `layers > 1`.
- This means a simple "merge consecutive pages into one `cudaMemcpyBatchAsync`
  descriptor" optimization is not available for the current per-layer direct
  path.  A GFD-style staging/gather path remains worth measuring because it can
  convert strided/scattered host page-layer fragments into one contiguous H2D DMA,
  at the cost of an extra CPU gather.

Correction / benchmark addition:

- Added `benchmark/nsa_prefill/benchmark_hicache_h2d_gfd_style.py` in
  tai-kernel.  It reports current descriptor count, physically mergeable direct
  descriptor count, staging gather fragment count, staging DMA descriptor count,
  host fixed-layer gap bytes, submit time, total time, and effective bandwidth.
- Added a pure-CPU unit test for the benchmark helpers to lock the key layout
  contract: with `layers=78`, contiguous source/destination page ids still have
  `direct_mergeable_descriptors == pages`; only a single-layer layout can merge
  fixed-layer page runs directly.
- The current `torch_staging` mode is intentionally a baseline/probe, not a
  production GFD implementation.  It uses PyTorch CPU gather into pinned staging
  plus one non-blocking H2D copy, so it should be interpreted as an upper-bound
  control-path prototype rather than as optimized AVX-512 polling-thread code.

Verification update:

- Local tai-kernel `py_compile`, `git diff --check`, and helper tests passed:
  `tests/nsa_prefill/test_hicache_h2d_gfd_style_benchmark.py` -> `4 passed`.
- Synced the new benchmark and tests to g0034.  Remote container helper tests and
  `py_compile` passed: `4 passed`.
- Remote CUDA smoke benchmark ran with `--tokens 1024 --payloads kv index
  --src-patterns contiguous random --dst-patterns contiguous --warmup 1
  --repeat 2`.  It produced CSV rows for `direct_batch` and `torch_staging` and
  confirmed the descriptor accounting column stays at `direct_mergeable_descriptors
  == current_descriptors` for the 78-layer page_first_direct layout.

Next measurement target:

- Run the benchmark with production-sized tokens (`4k, 8k, 10k, 40k, 120k`) and
  both BF16/FP8 payload settings after no inference job is using the remote GPUs.
- If optimized staging is still interesting after the baseline run, implement a
  dedicated tai-kernel/GFD-style CPU polling/gather path for index payload first,
  because index pages are much smaller and closer to the article's small-fragment
  workload than the main KV page payload.

C112 benchmark extension update:

- Extended `benchmark_hicache_h2d_gfd_style.py` with an `owner_lane` page pattern
  to model CP owner-lane page distribution.  For `cp_size=8`, `cp_rank=3`, the
  pattern produces `[3, 11, 19, 27, ...]`.
- Added `benchmark_hicache_h2d_multigpu_submit.py` as a thin multi-GPU entry
  point.  It can run either Python-thread workers or spawned-process workers,
  each bound to a CUDA device, and reports global wall time, per-worker measured
  wall time, submit latency, total latency, and aggregate bandwidth.
- The multi-GPU benchmark intentionally separates `global_wall_ms` from
  `worker_wall_max_ms`: process/thread startup and CUDA context initialization
  can dominate the former, while the latter is the useful transfer-window signal
  for driver-lock / submit-scaling analysis.

C112 verification update 2:

- Local tai-kernel helper tests passed: `6 passed`.
- Local `py_compile` passed for both benchmark entry points and the helper test.
- Synced benchmark/test files to g0034; remote container `py_compile` and helper
  tests passed: `6 passed`.
- Remote CUDA smoke ran:
  - single-GPU production-shape smoke for `4096` logical tokens with `kv/index`,
    `owner_lane/random`, `direct_batch/torch_staging`.
  - two-GPU thread smoke for `index/direct_batch/owner_lane`: produced CSV with
    `workers=2` and separate `global_wall_ms`/`worker_wall_max_ms` columns.
  - two-GPU process smoke for the same case: produced CSV and demonstrated that
    process startup/context init dominates `global_wall_ms`, so full production
    runs should compare `worker_wall_max_ms`, submit latency, and total latency.

C112 layout-compare benchmark update:

- Added `benchmark/nsa_prefill/benchmark_hicache_layout_compare.py` in
  tai-kernel to compare the current `page_first_direct` layout against a proposed
  `layer_page_first` layout without changing production code.
- The benchmark measures both H2D and D2H for a fixed layer.  It reports the
  current page_first_direct descriptor count and the descriptor count that would
  be possible if the host layout were `[layer, page, page_size, ...]`.
- For `contiguous` logical pages, `layer_page_first` reduces the descriptor count
  from pages to one run.  For `owner_lane` with CP stride, it does not coalesce
  unless the logical owner-lane pages are physically compacted/reordered in the
  host layout.

Remote smoke evidence on g0034:

- BF16, 4096 logical tokens, 64 pages:
  - contiguous H2D: page_first_direct `0.277 ms`, layer_page_first prototype
    `0.146 ms`.
  - contiguous D2H: page_first_direct `0.211 ms`, layer_page_first prototype
    `0.140 ms`.
  - owner_lane H2D/D2H: layer_page_first tensor-copy prototype is slower than
    page_first_direct because no page-run coalescing is possible and the
    prototype launches many small PyTorch copies.
- FP8 e4m3, 4096 logical tokens, 64 pages:
  - contiguous H2D: page_first_direct `0.245 ms`, layer_page_first prototype
    `0.103 ms`.
  - contiguous D2H: page_first_direct `0.171 ms`, layer_page_first prototype
    `0.104 ms`.
  - owner_lane again does not benefit without physical compaction.

Interpretation:

- `layer_page_first` can bring clear H2D/D2H speedup when the host pages for a
  fixed layer are physically contiguous or can be represented as long runs.
- In the current CP owner-lane distribution, simply changing `[page, layer, ...]`
  to `[layer, page, ...]` is not enough if the physical host page ids remain
  owner-lane strided.  To benefit in CP, we likely need either:
  1. compact owner-lane host allocation for each rank's local shard, or
  2. a transfer-time staging/gather path for owner-lane strided pages, or
  3. a new host layout that stores CP-owned logical page runs compactly per layer.
- Therefore the next decision should not be "switch all HiCache to
  layer_page_first" yet.  It should be: benchmark/implement a compacted
  `layer_page_first` owner-lane mapping, then compare against current direct
  page_first_direct under production token sizes.

C112 realistic-scatter benchmark gap:

- The first layout-compare smoke covered contiguous and owner-lane patterns, and
  the production-sized sweep covered fully random source pages.  Fully random
  pages are useful as a worst-case fragmentation bound, but real HiCache host
  pages are more likely to be a mixture of short contiguous extents separated by
  allocator/eviction gaps.
- To avoid overfitting the layout decision to either ideal contiguous pages or
  one-page random fragments, the benchmark should also include an
  allocator-fragmented pattern: random short page runs, preserving coalescing
  inside each run while scattering runs across the host pool.
- This pattern is especially relevant for comparing `page_first_direct` against
  a future `layer_page_first` host allocation.  If transferred pages remain
  fragmented by physical page id, layout reordering alone can regress; if the
  owner/rank-local shard is compacted into long runs, layer-first can reduce
  descriptors and improve H2D/D2H.

C112 realistic-scatter implementation / measurement update:

- Added `fragmented` page pattern to the tai-kernel benchmark helpers.  It
  generates random short contiguous page extents separated by fixed gaps, e.g.
  with `--fragment-run-pages 8 --fragment-gap-pages 8` a 3000-page transfer is
  represented as 375 coalescible runs instead of either one ideal run or 3000
  one-page random fragments.
- Remote g0034 helper verification passed after sync:
  `tests/nsa_prefill/test_hicache_h2d_gfd_style_benchmark.py -> 10 passed`.
- Remote 4k-192k sweeps were run for:
  - BF16 random source pages:
    `/mnt/beegfs/cjy/log/hicache_layout_compare_bf16_random_4k_192k.csv`
  - FP8 random source pages:
    `/mnt/beegfs/cjy/log/hicache_layout_compare_fp8_random_4k_192k.csv`
  - BF16 fragmented source pages:
    `/mnt/beegfs/cjy/log/hicache_layout_compare_bf16_fragmented_4k_192k.csv`
  - FP8 fragmented source pages:
    `/mnt/beegfs/cjy/log/hicache_layout_compare_fp8_fragmented_4k_192k.csv`

Key BF16 observations:

- Fully random pages are worst-case for non-compacted `layer_page_first`: 192k
  H2D is about `28.35 ms / 7.80 GB/s` for `layer_page_first same`, worse than
  current `page_first_direct` at `7.96 ms / 27.78 GB/s`.
- Fragmented 8-page extents are closer to allocator reality and show the useful
  middle ground: 192k BF16 H2D is `6.74 ms / 32.81 GB/s` for current
  `page_first_direct`, `5.52 ms / 40.09 GB/s` for `layer_page_first same`, and
  `4.50 ms / 49.18 GB/s` for compact `layer_page_first`.
- Compact `layer_page_first` remains the best BF16 path across 4k-192k:
  4k H2D improves from current `0.202 ms / 23.38 GB/s` to
  `0.132 ms / 35.86 GB/s`; 192k H2D improves from `6.74 ms / 32.81 GB/s` to
  `4.50 ms / 49.18 GB/s`.

Key FP8 observations:

- FP8 halves bytes moved, so fixed submit/descriptors matter more.  Fully random
  non-compacted `layer_page_first same` is very bad (`~3.9-4.0 GB/s`) because it
  degenerates into thousands of tiny copies.
- Fragmented FP8 still benefits from layer-first even without full compaction:
  192k H2D improves from current `5.43 ms / 20.38 GB/s` to
  `4.01 ms / 27.57 GB/s` with `layer_page_first same`, and to
  `2.53 ms / 43.71 GB/s` with compact `layer_page_first`.
- Therefore the production decision is not just "change memory dimension order".
  The safe performance contract is:
  1. if host pages stay fully random, keep current direct path or use a true
     gather/staging implementation;
  2. if allocator can preserve short extents, `layer_page_first` can already help;
  3. if CP owner/rank-local pages can be compacted per layer, `layer_page_first`
     gives the clearest H2D/D2H win for both BF16 and FP8.

### C113 — 2026-06-01 Current allocators do not actively optimize for contiguous host page runs

Finding:

- `HostKVCache.alloc()` is a simple FIFO slice of `free_slots`, initialized as a
  contiguous token range.  It preserves contiguity only while consuming the
  never-used tail or when previously freed pages happen to be queued contiguously.
  `HostKVCache.free()` appends returned indices to the end of `free_slots` and
  does not sort, coalesce, or search for the longest contiguous page run.
- CP device allocation is owner-lane aware, not contiguity aware.
  `CPSharedPagedTokenToKVPoolAllocator._select_compute_owner_pages()` selects the
  first available pages matching each requested owner lane and then replays the
  requested owner sequence.  Existing unit tests intentionally cover unsorted
  `free_pages` and selection from `release_pages` without sort-merge.
- Therefore the current production allocation behavior cannot guarantee the
  short contiguous host extents assumed by the `fragmented` benchmark pattern.
  Initial warm state can be contiguous, but after host eviction/release churn the
  host page order is driven by radix eviction/release order.

Implication:

- A `layer_page_first` production switch should not rely on today's allocator to
  naturally produce compact per-layer runs.  To realize the benchmarked compact
  path, we need an explicit compact/extent-aware host allocation policy or a
  transfer-time gather/staging path.

### C114 — 2026-06-01 Layout benchmark underestimates layer_page_first for fragmented pages

Correction:

- The current layout-compare benchmark is not a fair optimized-kernel comparison
  for non-contiguous `layer_page_first` pages.  The `page_first_direct` baseline
  uses the TAI direct `cudaMemcpyBatchAsync` path, while the `layer_page_first`
  prototype currently loops over Python/PyTorch `copy_()` calls for each page
  run.
- Therefore poor `layer_page_first same` results on fully random pages should be
  interpreted as "many small copies are bad" rather than "the layout is bad".
  `page_first_direct` is also affected by page fragmentation, but its benchmark
  path is already closer to production direct-batch submission.
- The layout-level signal remains valid: `page_first_direct` cannot make fixed
  layer adjacent pages physically contiguous because the host layout is
  `[page, layer, page_size, ...]`, while `layer_page_first` can collapse each
  contiguous page run to one descriptor because the host layout is
  `[layer, page, page_size, ...]`.

Next measurement requirement:

- Before deciding production layout, add an optimized TAI `layer_page_first`
  direct-batch path that builds one descriptor per contiguous page run.  Compare:
  current `page_first_direct`, optimized `layer_page_first same`, and compact
  `layer_page_first` under contiguous / fragmented / random page distributions.

### C115 — 2026-06-01 Added optimized TAI layer_page_first direct-batch kernel path

Implementation:

- Added TAI direct-batch per-layer APIs for proposed host layout
  `[layer, page, page_size, ...]`:
  - `transfer_kv_per_layer_direct_lf_lpf`: device layer-first LF -> host
    layer-page-first LPF (D2H backup).
  - `transfer_kv_per_layer_direct_lpf_lf`: host LPF -> device LF (H2D load).
- The new path keeps the same fail-fast CPU-index contract as
  `page_first_direct`: `src_indices` and `dst_indices` must be CPU int64
  contiguous and page-aligned. CUDA indices are rejected to avoid hidden D2H
  control-path copies.
- Descriptor construction now coalesces contiguous page runs where both source
  and destination token starts advance by exactly `page_size`; each run becomes
  one `cudaMemcpyBatchAsync` descriptor per logical KV tensor. This makes the
  `fragmented` benchmark contract meaningful: 8-page extents become 8
  descriptors instead of 64 for a 4k-token shard.
- `benchmark_hicache_layout_compare.py` now uses the new TAI direct APIs for
  `layer_page_first` instead of Python/PyTorch per-run `copy_()` loops.

Verification:

- Remote CUDA unit tests on `g0034` / `sglang-glm5-dev-2`:
  `PYTHONPATH=python python -m pytest -q tests/nsa_prefill/test_kvcacheio_lf_pf.py`
  passed: `35 passed in 2.44s`.
- Remote benchmark smoke with 4k fragmented pages, 8 layers, item size 96:
  - H2D `page_first_direct`: `0.388 ms`, `2.03 GB/s`.
  - H2D `layer_page_first same`: `0.104 ms`, `7.54 GB/s`.
  - H2D `layer_page_first compact`: `0.088 ms`, `8.90 GB/s`.
  - D2H `page_first_direct`: `0.133 ms`, `5.92 GB/s`.
  - D2H `layer_page_first same`: `0.100 ms`, `7.89 GB/s`.
  - D2H `layer_page_first compact`: `0.086 ms`, `9.12 GB/s`.
- Remote real-shape smoke with 4k fragmented pages, 78 layers, item size 576:
  - H2D `page_first_direct`: `0.276 ms`, `17.09 GB/s`.
  - H2D `layer_page_first same`: `0.176 ms`, `26.84 GB/s`.
  - H2D `layer_page_first compact`: `0.159 ms`, `29.68 GB/s`.
  - D2H `page_first_direct`: `0.205 ms`, `22.97 GB/s`.
  - D2H `layer_page_first same`: `0.169 ms`, `27.86 GB/s`.
  - D2H `layer_page_first compact`: `0.159 ms`, `29.61 GB/s`.

Remaining validation gap:

- This proves the optimized direct path works and removes the unfair Python-loop
  benchmark artifact. Full 4k-192k real-shape BF16/FP8 benchmarking is still
  needed before making `layer_page_first` the production default.

### C116 — 2026-06-01 Full LPF direct benchmark sweep after optimized direct path

Remote benchmark files on `g0034`:

- `/mnt/beegfs/cjy/log/hicache_layout_compare_lpf_direct_bf16_4k_192k.csv`
- `/mnt/beegfs/cjy/log/hicache_layout_compare_lpf_direct_fp8_4k_192k.csv`
- `/mnt/beegfs/cjy/log/hicache_layout_compare_lpf_direct_bf16_owner_lane_4k_40k.csv`
- `/mnt/beegfs/cjy/log/hicache_layout_compare_lpf_direct_fp8_owner_lane_4k_40k.csv`

Sweep parameters:

- `tokens=4k,10k,40k,120k,192k`, `layers=78`, `item_size=576`,
  `page_size=64`, `warmup=2`, `repeat=5`.
- Patterns: `contiguous`, `fragmented(run=8,gap=8)`, `random`.
- Additional CP owner-lane pattern was measured up to 40k.  Extending
  owner-lane to 192k requires physical page ids up to `3000*8`, which made the
  benchmark allocate about hundreds of GB of pinned host layout memory for both
  page-first and layer-page-first layouts.  That path was terminated and should
  be re-measured only after adding sparse/used-page host allocation to the
  benchmark.

Key 192k BF16 results, total median time:

| pattern | direction | page_first | LPF same | LPF compact | result |
|---|---:|---:|---:|---:|---|
| contiguous | H2D | 6.292 ms | 4.408 ms | 4.419 ms | LPF same 1.43x |
| contiguous | D2H | 6.196 ms | 4.408 ms | 4.408 ms | LPF same 1.41x |
| fragmented | H2D | 6.290 ms | 4.739 ms | 4.431 ms | LPF same 1.33x, compact 1.42x |
| fragmented | D2H | 6.226 ms | 4.724 ms | 4.428 ms | LPF same 1.32x, compact 1.41x |
| random | H2D | 6.315 ms | 6.326 ms | 4.398 ms | same neutral, compact 1.44x |
| random | D2H | 6.371 ms | 6.201 ms | 4.461 ms | same near-neutral, compact 1.43x |

Key 192k FP8 results, total median time:

| pattern | direction | page_first | LPF same | LPF compact | result |
|---|---:|---:|---:|---:|---|
| contiguous | H2D | 4.607 ms | 2.460 ms | 2.453 ms | LPF same 1.87x |
| contiguous | D2H | 4.520 ms | 2.616 ms | 2.451 ms | LPF compact 1.84x |
| fragmented | H2D | 4.612 ms | 2.737 ms | 2.450 ms | LPF same 1.68x, compact 1.88x |
| fragmented | D2H | 4.560 ms | 2.709 ms | 2.455 ms | LPF same 1.68x, compact 1.86x |
| random | H2D | 4.780 ms | 4.614 ms | 2.452 ms | same near-neutral, compact 1.95x |
| random | D2H | 4.857 ms | 4.506 ms | 2.447 ms | same near-neutral, compact 1.98x |

Owner-lane result at 40k:

- BF16 owner-lane `same` is neutral (`~1.0x`) because pages are strided by
  `cp_size=8`, so LPF cannot coalesce descriptors unless allocation is compacted.
  LPF compact is `~1.44-1.45x` faster.
- FP8 owner-lane `same` is also neutral (`~1.0x`); LPF compact is
  `~1.75-1.78x` faster.

Conclusion:

- Optimized LPF direct path is consistently better when physical host pages have
  contiguous extents.  The win is larger for FP8 because descriptor overhead is
  a larger fraction of total transfer time.
- LPF by itself does not fix fully random or owner-lane-strided host page ids;
  in those cases it is neutral unless we also compact/extent-aware allocate host
  pages or add a gather/staging path.
- Production switch should therefore be tied to host allocation policy: either
  compact CP owner-lane pages per layer, or keep page_first_direct for random
  residency and use LPF only when compact extents are guaranteed.

### C117 — 2026-06-02 Owner-lane benchmark models the CP allocator modulo-owner constraint, not full allocation history

Clarification:

- `owner_lane` in the benchmark is not a complete replay of production
  allocation history.  It isolates one important current allocator constraint:
  CP compute-owner pages are selected from modulo lanes.
- In production, `CPSharedPagedTokenToKVPoolAllocator._select_compute_owner_pages()`
  chooses pages for an owner where `(page_id - 1) % cp_size == owner`.
  Therefore one owner lane is physically strided by `cp_size`, for example
  `1, 9, 17, 25, ...` when `cp_size=8`.
- `build_in_seq_page_compute_owners()` generates a zigzag owner sequence for
  newly allocated pages.  The allocator then maps each owner in that sequence to
  the next available page from that owner's modulo lane.  Reuse from
  `free_pages` and `release_pages` can make the real layout more fragmented or
  random over time, but the modulo-lane stride is already enough to prevent
  LPF-same descriptor coalescing.

Implication:

- `layer_page_first same` is neutral on `owner_lane` because LPF only changes the
  per-page memory order; it does not make owner-lane physical page ids
  contiguous.  A fixed layer still sees many small descriptors when host page ids
  are `owner, owner+cp_size, owner+2*cp_size, ...`.
- Production LPF speedup requires changing allocation/residency policy, not only
  tensor layout: allocate compact/extent-aware host pages for each backed node or
  add a transfer-time gather/staging path.

### C118 — 2026-06-02 Chunked prefill can hit CP HiCache stale-tail split while backup is pending

Finding:

- Remote crash at `process_batch_result_disagg_prefill -> cache_unfinished_req -> insert -> _cp_prune_stale_tail_after_page_floor` raised `HiCachePendingBackupSplit` for a stale tail while CP HiCache backup was pending.
- The current insert path performs `_split_node()` first and only then calls `_cp_prune_stale_tail_after_page_floor()`.  If the stale tail subtree is unprunable because a backup lock/ref is pending, the exception is raised after the radix tree has already been mutated.
- `_split_node()` only checks `pending_host_backups`; the prune guard checks both `ongoing_write_through` and `pending_host_backups` via `_node_host_write_pending()`.  Therefore an ongoing-only write can pass the split precheck and fail during prune.
- `match_prefix()` catches `HiCachePendingBackupSplit` and defers scheduling, but `insert()` / `cache_unfinished_req()` do not convert this condition into a non-fatal deferred insertion.  Under chunked prefill the conflict can surface after forward, where requeueing the request is no longer possible.

Required correction:

- Make stale-tail split/prune atomic from the radix tree perspective: preflight pending/unprunable state before `_split_node()` and drain completed write acks once on the conflict path.
- If the stale tail is still pending, do not mutate the tree and do not crash the scheduler.  Defer/skip this cache insertion attempt while keeping request KV ownership intact for transfer/release.
- Do not delete the stale tail or split an in-flight backup node.  Per-layer backup ownership remains node/page based and cannot be repartitioned mid-flight.

### C119 — 2026-06-02 Prepared per-layer ack can arrive before radix attach under chunked prefill

Finding:

- Remote crash in `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260601_205025.log`
  was `KeyError: 54` in `HiRadixCache.writing_check()` rather than the earlier
  direct `HiCachePendingBackupSplit` exception.
- Sequence observed in the log:
  1. A chunked request skipped pre-forward backup with
     `[CP_HICACHE_FALLBACK][prepare_write_backup_skipped] reason=chunked_req`.
  2. A later chunk for the same request registered prepared per-layer backup for
     `node_id=54` and enqueued the final ack.
  3. A separate catch-up all-layer backup for `node_id=53` made
     `ongoing_write_through` non-empty.
  4. The stale-tail pending-split path called
     `_cp_drain_write_acks_before_pending_split() -> writing_check()`.
  5. `writing_check()` tried to pop ack id `54`, but `54` was not yet in either
     `pending_host_backups` or `ongoing_write_through` because the prepared
     backup had not been attached to a radix node yet.

Root cause:

- The old `writing_check()` contract assumed every ready ack in
  `ack_write_queue` already had corresponding radix state in
  `pending_host_backups` or `ongoing_write_through`.
- That assumption is false for prepared per-layer backups: the transfer can
  complete and enqueue its ack before `insert()` attaches the prepared backup to
  the newly inserted radix node.
- The bug is exposed by chunked prefill because chunked/catch-up writes can make
  `ongoing_write_through` non-empty while an unattached prepared ack is at the
  head of the queue.

Correction:

- `writing_check()` must only drain a ready ack entry when all `ack.node_ids` are
  registered in radix write state (`pending_host_backups` or
  `ongoing_write_through`).
- If a ready ack contains an unattached prepared node id, keep the ack queued and
  stop this drain pass.  Later `insert()` will either attach the prepared backup,
  making the ack committable, or rollback the unattached prepared backup and
  remove the orphan ack.
- Do not silently drop unknown ack ids; dropping would make a later attach wait
  forever for an ack that no longer exists.

Regression test:

- `test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_writing_check_defers_unattached_prepared_ack`
  constructs an ack queue with an unattached prepared ack `[54]` ahead of an
  attached catch-up ack `[53]`.  `writing_check()` must not raise `KeyError`, must
  preserve both ack entries, and must not release the attached node behind the
  unattached head entry.

### C120 — 2026-06-02 Chunked prefill should cache completed middle-chunk pages

Finding:

- With chunked prefill enabled, the CP HiCache path previously skipped
  pre-forward prepared backup for every chunked request:
  `[CP_HICACHE_FALLBACK][prepare_write_backup_skipped] reason=chunked_req`.
- `cache_unfinished_req(chunked=True)` still inserted scheduler-visible valid
  tails.  For CP HiCache this is the wrong granularity: the cache is managed at
  page granularity, so inserting a non-page-aligned chunk tail creates a stale
  tail that a later chunk must prune while the same request can still own or
  lock those KV slots.
- Remote fallback storms showed this exact shape: many
  `prepare_write_backup_skipped` and `insert_deferred_pending_backup_split`
  events, often with `ongoing=0 pending=0`, meaning the split was blocked by
  node protection/refs rather than an actual active transfer.

Requirement:

- Intermediate chunks are not disposable.  They should write completed pages
  into CP HiCache to maximize reuse and avoid doing all host backup only at the
  final chunk.
- The current incomplete page must remain request-owned until a later chunk or
  final insert makes it page-complete.  CP HiCache should not create a radix
  node for a sub-page valid tail.

Correction:

- `prepare_write_backup_for_req()` no longer skips chunked requests.  For
  chunked CP HiCache requests it floors the backup end to the nearest completed
  page and reserves/writes only `[start, floor_to_page_len(len(keys)))`.
- `cache_unfinished_req(chunked=True)` now inserts only completed pages when CP
  HiCache is active.  If no new completed page exists beyond
  `cache_protected_len`, it keeps `req.prefix_indices` pointing at the full
  request-owned KV span and returns without mutating the radix tree.
- The sub-page tail is deliberately retained in `req.prefix_indices` but not in
  the CP HiCache radix tree.  This trades at most one page of temporary
  non-cacheable tail for stable page-granular radix/host-cache state.

Regression tests:

- `test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_prepare_write_backup_for_req_chunked_reserves_completed_pages`
  verifies a 10-token chunk with `page_size=4` reserves only the first 8
  completed tokens instead of warning and skipping backup.
- `test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_cp_cache_unfinished_chunked_inserts_only_completed_pages`
  verifies `cache_unfinished_req(chunked=True)` inserts only 8 page-completed
  tokens while preserving full 10-token `req.prefix_indices` for the scheduler.

Validation:

- Remote targeted regression tests on `g0034`:
  `2 passed, 3 warnings`.
- Remote full metadata suite:
  `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py`
  passed with `107 passed, 5 warnings`.

Remaining validation gap:

- Requires a fresh ETE run with chunked prefill enabled to verify that the
  fallback storm disappears in production traffic.  An already-running remote
  process will not pick up this change until restarted.

### C121 — 2026-06-02 CPU overhead must be measured by microbench, not inferred from logs

Finding:

- Runtime logs can show fallback storms and fatal paths, but they cannot rank normal-path CPU overhead.  The hot paths here are often successful allocator/control-path calls that produce no warning log.
- The CP shared-KV L1 owner-lane allocator had two visible metadata costs:
  - `compute_owner_lane_stats()` scanned the full free-page tensor once per CP owner and performed one `.item()` synchronization per owner.
  - `_select_compute_owner_pages()` recomputed owner masks per owner and built prefix selections via full-tensor `cumsum`, then optionally searched contiguous owner-lane runs.
- Host/L2 `alloc_contiguous_preferred()` is also expensive on fragmented metadata because it validates page contiguity over the full free-slot tensor and searches for later runs.  A local 210GB-equivalent metadata benchmark showed fragmented/random contiguous-preferred allocation at roughly 8–11 ms p50 versus FIFO at sub-ms scale for the same request sizes.

Correction implemented:

- Extended `benchmark/hicache/bench_cp_hicache_allocator_overhead.py` with L1 operation breakdowns:
  - `stats`
  - `free_room_stats`
  - `select_only`
  - `alloc_pages`
- Added a dependency-light standalone CP shared-paged allocator model so the CPU metadata shape can be measured locally without importing the full SGLang runtime stack.
- Optimized the production CP owner-lane stats path from per-owner mask+sum scans to one `torch.bincount()` over page owners.
- Optimized owner-page selection by computing free/release page owners once per allocation and replacing the prefix `cumsum` selection with `nonzero()[:required_count]` prefix selection.
- Replaced contiguous-run detection from `run_edges.unfold(...).all(dim=1)` with a cumulative-sum sliding-window test.  This preserves the same contiguous-run contract but avoids O(num_free_pages * required_pages) metadata work for large requests.

Current evidence:

- Local CPU benchmark after the stats/path cleanup still shows selection/allocation as the dominant cost.  Example focused run at `physical_pages=32768`, `cp_size=8`, random/zigzag owners:
  - `stats`: about 0.9–1.4 ms p50.
  - `select_only`: about 8–12 ms p50 for the focused random/fragmented zigzag cases.
  - `alloc_pages`: about 8–12 ms p50 for the same cases, request sizes 64–512 pages.
- Therefore the first optimization only reduces part of the overhead; the main remaining cost is still owner-lane page selection, especially contiguous-preferred search and full-size mask materialization.

Verification:

- Local:
  - `python -m py_compile python/sglang/srt/mem_cache/allocator.py benchmark/hicache/bench_cp_hicache_allocator_overhead.py test/registered/unit/benchmark/test_cp_hicache_allocator_bench.py`
  - `PYTHONPATH=. python -m pytest -q test/registered/unit/benchmark/test_cp_hicache_allocator_bench.py` → `6 passed, 1 warning`.
- Remote `g0034` container:
  - benchmark unit subset: `6 passed, 1 warning`.

Known verification gap:

- Remote `test_cp_shared_kv_layout.py` collection aborted while importing `sgl_kernel` native ops in the container, before reaching the allocator tests.  This is an environment/native import failure during test collection, not evidence of allocator logic failure.
- Need a clean remote production-allocator test path or a container state where `sgl_kernel` imports safely before claiming full production allocator verification.

Next target:

- Replace full-tensor owner-lane selection with a lower-overhead data structure or batched selector.  The viable directions are:
  1. Maintain per-owner free-page queues/counters incrementally.
  2. Use a bounded contiguous-search policy and skip expensive run search for small requests or highly fragmented pools.
  3. Add a fused selector kernel if GPU-side selection remains acceptable, but avoid increasing synchronization frequency.

C121 additional remote standalone benchmark evidence:

- `g0034` container, dependency-light standalone allocator model, `physical_pages=32768`, `cp_size=8`, zigzag owners:
  - random, 64 pages: `stats` p50 0.87 ms, `select_only` p50 6.55 ms, `alloc_pages` p50 7.29 ms.
  - owner-fragmented later-run, 64 pages: `stats` p50 0.43 ms, `select_only` p50 6.13 ms, `alloc_pages` p50 6.80 ms.
  - random, 512 pages: `stats` p50 0.90 ms, `select_only` p50 7.46 ms, `alloc_pages` p50 7.91 ms.
  - owner-fragmented later-run, 512 pages: `stats` p50 0.45 ms, `select_only` p50 6.89 ms, `alloc_pages` p50 7.16 ms.
- This reinforces that after the cheap cleanup, stats is no longer the main bottleneck; selector/allocation metadata still costs multi-ms and is the next CPU-overhead target.

### C122 — 2026-06-02 L1 owner-lane allocation must remove pure-CPU full scans

Finding:

- A remaining ~7–10 ms allocator/control-path cost is still too high because it
  is pure CPU metadata work.  Unlike D2H/H2D backup/load kernels, this work
  cannot overlap with GPU forward progress once the scheduler is blocked waiting
  for page allocation.
- The C121 cleanup made stats cheaper, but selection still scanned or
  materialized full free-page tensors on every owner-lane allocation.
- This shape scales with total cache capacity, not request size.  With a 150–220
  GB HiCache/L1 metadata scale, even a successful hot-path allocation becomes a
  scheduler stall.

Correction in progress:

- Make `CPSharedPagedTokenToKVPoolAllocator` keep per-owner free/release page
  buckets as allocator state instead of deriving owner buckets by scanning
  `free_pages` for every allocation.
- Keep allocation itself request-sized: count required pages per owner, take the
  needed prefix from that owner bucket, and consume bucket prefixes only after
  all lanes are known satisfiable.
- Preserve the public `free_pages` / `release_pages` tensor interface through
  lazy materialized caches for tests and legacy paths, but avoid materializing it
  in the CP owner-lane fast path.
- Sort within each owner bucket when pages are inserted/restored.  This changes
  the CP owner allocator from global FIFO semantics to owner-lane contiguous
  semantics, which is more aligned with the RDMA/H2D/D2H goal: each owner lane
  should prefer physically consecutive pages without a per-allocation run scan.

Risk / contract note:

- Code that only depends on owner-correct page allocation is unaffected.
- Code or tests that implicitly relied on exact global `free_pages` order after a
  CP owner-lane allocation must be updated; global FIFO ordering is not the
  intended CP shared-KV owner-lane contract.

C122 validation update:

- Remote `g0034` production allocator microbench with `--stub-sgl-kernel`,
  `physical_pages=32768`, `cp_size=8`, zigzag owners:
  - random, 64 pages: `stats` p50 17 us, `select_only` p50 173 us,
    `alloc_pages` p50 527 us.
  - owner-fragmented later-run, 64 pages: `stats` p50 13 us,
    `select_only` p50 166 us, `alloc_pages` p50 235 us.
  - random, 512 pages: `stats` p50 27 us, `select_only` p50 284 us,
    `alloc_pages` p50 638 us.
  - owner-fragmented later-run, 512 pages: `stats` p50 32 us,
    `select_only` p50 266 us, `alloc_pages` p50 476 us.
- This confirms the allocator hot path is no longer full-cache-scan shaped for
  this benchmark: the previous 6–8 ms selector/allocation p50 is reduced to
  sub-ms p50 while preserving owner-lane correctness and contiguous lane
  selections.
- Remaining CPU cost is now mostly request-sized construction (`page_compute_owners`
  grouping, output page tensor fill, and token-loc expansion), not total-cache
  sized metadata scans.

### C123 — 2026-06-02 CPU allocator tests must not load native sgl_kernel during collection

Finding:

- Remote `test_cp_shared_kv_layout.py` aborted during pytest collection inside
  `sgl_kernel/load_utils.py::_load_architecture_specific_ops` before any test
  logic ran.
- This is not an allocator correctness failure.  The test imported
  `sglang.test.test_utils`, which imports `sglang.srt.utils.common`; that module
  imports `sgl_kernel` to probe AMX availability.  On the remote image the native
  loader aborts the process instead of raising a catchable Python exception.
- A `try: import sgl_kernel` fallback is insufficient for this environment
  because SIGABRT bypasses Python exception handling.

Correction:

- Install a minimal `sgl_kernel`, `sgl_kernel.kvcacheio`, and
  `sgl_kernel.quantization` stub at the top of CPU-only allocator/layout tests
  before importing any `sglang` helper module.
- Keep Torch custom-op schema registration for the operators referenced by the
  imported SGLang code, but do not load the native extension during collection.

Scope:

- This applies only to CPU unit tests.  It does not change production runtime
  import behavior and does not mask native kernel problems in CUDA/ETE tests.

C123 validation update:

- After avoiding the native import abort, collection reached Python test logic but
  failed because the CPU stub had not defined the fp8 quantization custom-op
  schemas used by `fp8_kernel.py` fake registrations.
- The test stub now defines the fp8 quantization and fp8 GEMM schemas before any
  `sglang` import, matching the CPU-only pattern used by the heavier HiCache
  metadata tests.

C123 full-suite update:

- The full `test_cp_shared_kv_layout.py` file then reached scheduler rollback
  tests and failed because `memory_pool_host.py` imports named transfer helpers
  from `sgl_kernel.kvcacheio`.
- The CPU stub now also gives `sgl_kernel.kvcacheio` a module-level
  `__getattr__`, so named imports resolve to inert functions without importing
  the native extension.

### C124 — 2026-06-02 L2 host churn benchmark must model fragmented steady-state eviction

Finding:

- The earlier host allocator benchmark measured one cold allocation against a
  synthetic `free_slots` layout.  That is not enough for production HiCache: L2
  host cache can be ~220 GB, runs near full occupancy, and repeatedly frees old
  nodes before reserving a new node.
- If prefill nodes and measured allocation requests have the same page size, a
  random eviction workload can still produce unrealistically contiguous freed
  chunks.  This hides the allocator/pathology we care about for layer-page-first
  (LPF) transfer planning.

Correction:

- Added `benchmark/hicache/bench_cp_hicache_allocator_overhead.py --bench
  host_churn` for steady-state L2 metadata churn.
- The benchmark now records both CPU allocation latency and transfer-descriptor
  quality proxies:
  - `pf_desc`: page-first descriptor count, one descriptor per selected page.
  - `lpf_desc_mean` / `lpf_ratio`: layer-page-first descriptor count after
    coalescing consecutive physical page runs.
  - `run_p50` / `max_run_mean`: selected physical-page run quality.
- Added `--host-churn-prefill-node-pages` so the filled resident set can use a
  different historical node size than the new allocation request.  This models
  cache built from many small chunks, then later serving larger extensions.

Scope:

- This is CPU metadata-only.  It does not allocate real 220 GB KV buffers and it
  does not test CUDA kernel bandwidth.
- It is intended to decide whether L2 `HostKVCache.alloc_contiguous_preferred()`
  / future L2 bucket allocator work is on the actual hot path, and whether LPF
  layout can get enough physical-page coalescing to help H2D/D2H/RDMA.

C124 validation update:

- Local CPU unit coverage: `test_cp_hicache_allocator_bench.py` passes with the
  fragmented prefill-node churn case.
- Remote `g0034` container validation: py_compile passed and the benchmark unit
  file passed (`9 passed`).
- Remote 220GB-equivalent production `HostKVCache` churn sample
  (`pages=34375`, `occ=0.97`, `request_pages=512`, `evict_pages=512`):
  - `prefill_node_pages=1`: FIFO p50 ~11 us but LPF run quality is poor
    (`lpf_ratio≈0.663`, `run_p50≈505`).  Contiguous-preferred p50 ~700 us and
    still cannot find a good run under this fragmentation.
  - `prefill_node_pages=64`: FIFO p50 ~9 us and LPF run quality is much better
    (`lpf_ratio≈0.012`, `run_p50≈8`).  Contiguous-preferred p50 ~499 us.
- Interpretation: current production `alloc_contiguous_preferred()` can add
  hundreds of microseconds on fragmented 220GB-equivalent metadata.  The
  benchmark now makes this measurable separately from ETE noise; it also shows
  that node-size history strongly affects LPF transfer coalescing potential.

### C125 — 2026-06-02 Host churn benchmark needs explicit burn-in to avoid cold free-tail bias

Finding:

- A high-occupancy prefill still leaves an initial contiguous free tail.  With
  small `warmup` and small requests, measured allocations can consume this cold
  tail before they ever allocate from evicted fragmented nodes.
- That can overstate LPF run quality and understate the allocation/search cost we
  expect after the service has churned for a while.

Correction plan:

- Add a separate `--host-churn-burnin` iteration count.  Burn-in iterations run
  the same evict+allocate cycle but are not measured and are independent of
  benchmark warmup.
- Use burn-in to exhaust the initial free tail before collecting steady-state
  latency and run-quality samples.

C125 validation update:

- Local RED/GREEN: `test_host_churn_burnin_exposes_fragmented_evicted_nodes_after_cold_tail`
  first failed on missing `burnin`, then passed after adding the burn-in path.
- Local full benchmark unit file: `10 passed` plus py_compile.
- Remote `g0034` container: py_compile passed and
  `test_cp_hicache_allocator_bench.py` passed (`10 passed`).
- Remote 220GB-equivalent production `HostKVCache` sample with `burnin=20`,
  `occ=0.97`, `evict_pages=512`:
  - `request_pages=64`, `prefill_node_pages=1`: FIFO p50 ~11 us but LPF quality
    is worst case (`lpf_ratio=1.000`, all single-page runs).  The contiguous
    search can cost ~2.6 ms p50.
  - `request_pages=512`, `prefill_node_pages=1`: FIFO p50 ~10 us, LPF quality
    remains worst case (`lpf_ratio=1.000`).  Contiguous search p50 ~576 us and
    cannot improve run quality because no 512-page run exists.
  - `prefill_node_pages=64`: LPF quality is much better (`lpf_ratio≈0.012–0.031`)
    but contiguous search can still cost ~0.5–1.5 ms p50.
- Interpretation: this benchmark now exposes the relevant tradeoff: FIFO is cheap
  but can create very high descriptor count for LPF/RDMA when historical nodes
  are tiny; naive contiguous search can be milliseconds on 220GB-equivalent host
  metadata.  A future L2 allocator should avoid full free-list scans and preserve
  larger free extents/buckets rather than searching linearly per allocation.

### C126 — 2026-06-02 L2 HostKVCache allocator uses lazy page extents instead of full free-slot scans

Finding:

- `HostKVCache.alloc_contiguous_preferred()` previously scanned/materialized the
  full token-level `free_slots` tensor to discover contiguous physical page runs.
  On 220GB-equivalent host metadata this could cost hundreds of microseconds to
  milliseconds per reservation.
- The full scan is the wrong shape for CP HiCache: allocations and releases are
  page-shaped, and the fast path only needs page-run metadata plus a token-index
  tensor for the chosen pages.

Correction:

- `HostKVCache` now maintains a lazy page-extent index:
  - `free_slots` remains a compatibility property and is materialized lazily only
    when external code reads it.
  - `available_size()` reads an integer token count and does not materialize.
  - `free()` converts page-shaped token indices into page runs and merges them
    into sorted extents.
  - `alloc_contiguous_preferred()` first checks the largest free extent for a
    single-run allocation; if no run is large enough, it falls back to a batched
    fragmented run allocation without scanning/sorting the whole `free_slots`
    tensor.
- The fallback is still page-shaped and returns normal token indices.  It avoids
  silent corruption: overlapping/double-free page extents raise, and non-page
  shaped frees disable the extent index with a warning before falling back to the
  legacy tensor path.

C126 validation update:

- Remote RED/GREEN target: the new lazy extent-index unit first failed on missing
  `_free_slots_dirty`, then passed after implementation.
- Remote `test_hicache_controller_cp.py`: `67 passed`.
- Remote `test_cp_hicache_allocator_bench.py`: `10 passed`.
- Remote 220GB-equivalent production `HostKVCache` churn sample with `burnin=20`,
  `occ=0.97`, `evict_pages=512` after optimization:
  - `request_pages=64`, `prefill_node_pages=64`: contiguous p50 ~31 us.
  - `request_pages=512`, `prefill_node_pages=64`: contiguous p50 ~86 us.
  - `request_pages=512`, `prefill_node_pages=1`: contiguous p50 ~292 us; this
    is no longer a full free-list scan, but mostly fragmented selection plus
    constructing a 32768-token host-index tensor.
- Compared with the previous C125 sample, the contiguous path drops from roughly
  0.5–2.6 ms p50 to roughly 30–292 us p50 on this benchmark shape.

Remaining risk:

- `alloc()` now uses the extent-backed fragmented allocation when the extent
  index is active, so exact FIFO order is no longer the internal contract.  This
  should be acceptable for host KV slots because callers require unique free
  page-shaped slots, not FIFO identity, but future code must not depend on
  token-level FIFO order.
- Very fragmented tiny historical nodes still produce poor LPF coalescing.  That
  requires higher-level eviction/allocation policy to preserve larger free runs;
  the allocator now avoids the worst CPU scan cost but cannot create physical
  contiguity that the free set does not contain.

### C127 — 2026-06-02 Disabled index prefetch must not emit fallback warnings

Finding:

- `CpSharedKVIndexPrefetcher.maybe_create()` correctly returns `None` without a
  warning when `SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH=0`.
- The later synchronous materialization path in `nsa_indexer.py` still logged
  `[CP_SHARED_KV_FALLBACK][index_prefetch] reason=missing_prefetcher` whenever
  `forward_batch.cp_shared_kv_index_prefetcher` was `None`.
- With MLA/index prefetch disabled, `None` is the expected state, so this warning
  made normal sync materialize look like a failed fast path and spammed logs.

Correction:

- Gate only the `missing_prefetcher` / `current_missing_prefetcher` warnings on
  `cp_shared_kv_mla_prefetch_enabled()`.
- Keep `consume_miss` warnings unchanged: if a prefetcher exists but cannot serve
  the requested layer/current buffer, that is still a real enabled-prefetch miss.

Validation:

- Remote RED/GREEN test added:
  `test_index_sync_materialize_does_not_warn_when_prefetch_disabled`.
- Remote targeted regression set passed (`4 passed`): disabled sync materialize,
  consume miss warning, first-layer no-warning, and create-skip warning when
  enabled.
