# NSA Prefill CP shared-KV：accept len=1 / 输出乱码排查 Ledger

日期：2026-06-04

目标：记录已经检查过的证据、排除项和当前假设，避免后续在日志/代码压缩后重复阅读同一批路径。

维护规则：

- 后续每确认一个新事实，先追加到本文，再继续排查。
- 只记录有日志、代码位置或测试输出支撑的事实；推测必须放在“假设”段落，并标注待验证。
- 已排除项不要在后续轮次重新当作主线，除非出现新的相反证据。
- 修复前必须先把“哪个 logical/physical/padded 合同被破坏”定位清楚。

## 0. 当前现象

用户在 g0033 经 router 调用：

```text
python summarize_story.py
api_url=http://g0034:17000/v1/chat/completions
model=GLM5
prompt_tokens=40387
completion_tokens=113
HTTP_OK seconds=5.79
输出大量无意义 token / 重复 </think>
```

同时 decode 日志显示 accept len 掉到约 1.0。

## 1. 当前远端运行配置已确认

时间：2026-06-04 07:51 CST 左右。

prefill 进程在 `g0034`：

```text
python -m sglang.launch_server ...
--disaggregation-mode prefill
--tp-size 8 --ep-size 8
--enable-nsa-prefill-context-parallel
--nsa-prefill-cp-mode in-seq-split
--attn-cp-size 8
--enable-nsa-prefill-cp-shared-kv
--speculative-algorithm EAGLE
--speculative-num-steps 3
--speculative-num-draft-tokens 4
--enable-hierarchical-cache
--hicache-size 150
--hicache-io-backend direct
--hicache-mem-layout page_first_direct
--kv-cache-dtype fp8_e4m3
--enable-cp-shared-kv-prefill-bs-gt1
--cp-shared-kv-prefill-max-batch-requests 5
--cp-shared-kv-prefill-max-total-extend-tokens 65536
--chunked-prefill-size 65536   # 命令里有 -1 和 65536，ServerArgs 生效为 65536
```

关键环境变量由用户命令提供：

```text
SGLANG_EAGLE_ACCEPT_DEBUG=1
SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH=1
SGLANG_CP_DRAFT_SHARED_KV=1
SGLANG_CP_SHARED_KV_FUSED_INDEX_MQA_PREPARE=1
SGLANG_CP_SHARED_KV_FUSED_MLA_STORE=1
SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH=1
SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1
SGLANG_CP_SHARED_KV_CURRENT_REUSE=1
SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1
```

decode 在 g0035/g0036，日志文件当前为：

```text
/mnt/beegfs/cjy/log/decode0_20260603_220439.log
/mnt/beegfs/cjy/log/decode1_20260603_220442.log
```

prefill 最新日志：

```text
/mnt/beegfs/cjy/log/sglang_cp_hicache_20260603_232444.log
```

## 2. 本次坏请求对应日志证据

### 2.1 prefill 请求与 cache 写入

prefill 日志中坏请求：

```text
[2026-06-03 23:28:20] Requested token count (40391 input + 200000 new) exceeds ... Truncating max_new_tokens.
```

随后 HiCache backup：

```text
[23:28:21 ATTN_CP0..7] [CacheCtrl-write] submit_write_cp_per_layer registered: node_id=4 logical_len=40386 layers=78 draft=True
[23:28:21 ATTN_CP0..7] [HiCache-write] prepared CP per-layer backup before forward: ... logical_len=40386 owned_positions=5056
[23:28:22 ATTN_CP0..7] [CacheCtrl-write] submit_write_cp_layer final ack: node_id=4 logical_len=40386 layers=78 draft=True
```

注意：日志中的 request input 是 40391，但 HiCache node logical_len 是 40386。该差异尚未定性，不能重复忽略。

### 2.2 prefill 实际 forward token 数

```text
[23:28:22 ATTN_CP0..7] Prefill batch, #new-seq: 1, #new-token: 40448, #cached-token: 0
```

解释：`40448 = ceil(40386 / 64) * 64`，这是 page granularity / static padding 后的物理计算 token 数。它本身不是 bug 证据；bug 应该出在某条路径把物理 padding 当成 valid/logical token 消费。

### 2.3 index current reuse fallback 警告

同一请求中出现大量：

```text
[CP_SHARED_KV_FALLBACK][tai_index_mqa_prepare] reason=current_batch_kernel_failed
error=current_index_k must be uint8 segments=2 total_kv_len=43072..43456 total_q_count=5056
```

已知事实：

- 这些 warning 主要出现在 CP1-CP7；CP0 的长请求 warning 未观察到。
- 对 `L=40386, page_size=64, cp_size=8`：
  - padded tokens = 40448
  - CP1-CP7 的 valid local rows = 5056，因此 `total_q_count=5056` 对 CP1-CP7 本身不证明 tail 泄漏。
  - CP0 valid local rows 应为 4994，full-page compute rows为 5056。
- 所以该 warning 目前更像 dtype/perf fallback：TAI current-index prepare 期望 `uint8`，而调用侧可能传入 `float8_e4m3fn` view；不应直接把它当成乱码根因。

### 2.4 decode accept len

decode1 当前请求：

```text
[23:28:24 DP15] Decode batch, #running-req: 1, #token: 40448, accept len: 1.10, accept rate: 0.28
[23:28:25 DP15] Decode batch, #running-req: 1, #token: 40512, accept len: 1.00, accept rate: 0.25
```

说明：`#token=40448` 来自 allocator/token usage 统计，不一定等于 logical seq len；不能仅凭这条判定 decode 使用了 padding token，但它与 prefill 物理 page padding 数一致，需要继续追踪消费者是否使用了 `len(kv_loc)`。

## 3. 已排除/不要重复误判的点

### 3.1 不是“服务没同步/旧代码”导致的直接结论

之前已通过 md5 对比确认远端关键文件与本地一致：

```text
nsa_indexer.py md5 efe7bcc118fbd4cc84c6017aac818228
utils.py       md5 b953059a79a7e19cd12154098d995ada
```

当前进程启动时间在最新 commit 之后，且日志中已经出现最新 fail-fast/fallback 字符串。

### 3.2 早期 500 / OUTPUT_ZERO 主要是 prefill bootstrap 不可用

decode 日志中 22:45-23:18 大量：

```text
Could not fetch prefill parallel info from g0034:18992
Decode transfer failed ... Losing connection with prefill instance
[OUTPUT_ZERO_DEBUG] ... finish_reason abort
```

这些发生在 prefill 重启/不可用阶段，不是当前 23:28 HTTP 200 但输出乱码的主证据。后续不要把这些旧 500 当成当前乱码根因。

### 3.3 warmup 输出也不正常，但不等同本次根因

decode warmup response 有：

```text
text='! of the of the of the'
spec_accept_rate=0.25
spec_accept_length=2.0
```

这说明模型/EAGLE 在短 warmup 上也偏异常，但短 warmup 与 40k prompt + CP shared KV + FP8 + HiCache 的路径不同。可作为背景信号，不能单独定位 root cause。

## 4. 已阅读的关键代码路径

本节用于防止重复读同一批入口。后续如果继续深入某个函数，需要在对应条目下补“确认/排除/风险”。

### 4.1 CP metadata / split / local cache loc

文件：`python/sglang/srt/layers/attention/nsa/utils.py`

已看函数/对象：

- `CPSharedKVBatchPlan`
- `build_batch_page_aligned_in_seq_split_plan()`
- `split_tensor_by_cp_batch_plan()`
- `_get_cp_local_valid_row_indices_cache()`
- `select_cp_local_valid_rows_for_cache_write()`
- `get_cp_shared_kv_local_out_cache_loc()`

当前理解：

- 普通 page-tail padding 与 compute-padding 是两个概念。
- 现在 `compute_padding_enabled` 只代表“为了避免 tiny/zero lane，把 compute padded 到至少 cp_size pages”等更强 padding；长请求尾页 padding 不一定使它为 true。
- 因此任何只在 `compute_padding_enabled=True` 时才裁 valid rows 的路径，都需要审查是否会在普通尾页下泄漏 padding。

当前状态：

- 已确认：`40448` 这类 full-page token 数可以是合法物理覆盖。
- 未确认：所有 current-only reuse 路径是否都使用 CP-local valid rows，而不是 global valid rows 或 physical padded rows。

### 4.2 NSA index current reuse/top-k

文件：`python/sglang/srt/layers/attention/nsa/nsa_indexer.py`

已看函数：

- `_maybe_materialize_shared_index_buffer()`
- `_get_topk_ragged_with_cp()`
- `_get_topk_in_seq_cp_pair_batch()`
- `forward_cuda()` 中 current index reuse / cache write 逻辑

关键点：

- current-only 时 `_get_topk_in_seq_cp_pair_batch()` 不 materialize prefix，而是把 `current_index_kv_for_topk` 传给 `_get_topk_ragged_with_cp()`。
- `_select_batch_topk_query_lengths()` 已用于根据 `q_fp8/weights` 实际行数选择 valid/compute metadata，修复过 40392 vs 40387、5056 vs 4995 类 fail-fast。
- 现在仍需确认 current-only 下 `kv_len`、`seq_len`、`logical_kv_limit` 与 current rows 是否完全一致。

当前状态：

- 已确认：FP8 下 `tai_index_mqa_prepare` warning 是 `current_index_k must be uint8`，至少包含 dtype fast-path 不匹配。
- 未确认：fallback 后 Python 拼接路径是否仍严格保持 index/top-k 的 logical order 与 CP owner-lane order。

### 4.3 MLA current/partial-current reuse

文件：`python/sglang/srt/layers/attention/nsa_backend.py`

已看区域：

- `init_forward_metadata()` / `_transform_table_1_to_real()`
- MLA current-only / partial-current reuse around `_forward_flashmla_kv`

当前高优先级怀疑：

- current-only MLA reuse 通过 `materialize_prefix_and_reuse_current_kv_page_slots(prefix_pages=0)` 构造 page-slot dense KV。
- 如果 `logical_page_table_1` 覆盖的 logical loc 数量大于 `current_locs_for_reuse` 覆盖数量，就会出现部分 logical page/loc 没有 current KV 填充，但 attention 仍可能消费。
- 这类问题对 FP8 更敏感，因为 FP8 cache 必须严格 page-slot aligned，不能 compact。

当前状态：

- 未确认：`logical_page_table_1` 的消费范围是否只覆盖 valid logical loc。
- 未确认：FP8 page-slot compose 与 BF16 page-slot compose 是否走完全相同的 logical/page-table 合同。

追加确认（2026-06-04）：

- `forward_mla.py::_maybe_write_cp_shared_local_mla_kv()` 在 `rebuild_cp_kv_cache()` 之前写 persistent CP shared-KV。
- 当 `should_reuse_current_extend_kv()` 为真时，prefill attention 本轮可以通过 `rebuild_cp_kv_cache()` 得到 all-gather 后的 current KV；这条路径与写入给 decode/HiCache/transfer 使用的 persistent KV 不是同一个消费者。
- 因此，`prefill HTTP 200` 不能证明 persistent KV 写入正确；当前乱码更可能是 persistent KV 写入或 transfer 映射损坏，decode 消费后 EAGLE accept len 掉到 1。

### 4.4 runtime helpers

文件：`python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py`

已看函数：

## 5. 2026-06-05 SSH 恢复后的远端证据补充

### 5.1 最新 cjy decode 进程已不在，且 decode0 日志无 Python fatal

检查时间：2026-06-05 17:10 CST 左右。

远端 `g0035/g0036/g0033` 上当前活跃的 `sglang.launch_server` 是其他 workload：

```text
python3 -u -m sglang.launch_server ... --port 30000 ... --disaggregation-mode prefill ...
```

不是本轮 cjy decode 进程。共享日志目录最新 cjy 相关文件仍是：

```text
/mnt/beegfs/cjy/log/decode0_20260605_003305.log
/mnt/beegfs/cjy/log/sglang_cp_hicache_20260605_003303.log
```

`decode0_20260605_003305.log` 中：

- decode server args 明确是 `disaggregation_mode='decode'`；
- `enable_nsa_prefill_context_parallel=False`；
- `enable_nsa_prefill_cp_shared_kv=False`；
- `enable_cp_shared_kv_prefill_bs_gt1=False`；
- `enable_hierarchical_cache=False`；
- `kv_cache_dtype='fp8_e4m3'`；
- `speculative_algorithm='EAGLE'`；
- `nsa_decode_backend='flashmla_kv'`。

所以 decode 端不直接启用 CP shared-KV/bs>1；它仍会消费 prefill transfer 过来的 KV/metadata。

同一日志没有出现：

```text
Traceback / Scheduler hit / SIGQUIT / Health check failed / FAIL_FAST / Metadata corruption / Decode transfer failed
```

文件最后停在 00:39:54 多条 `POST /v1/chat/completions HTTP/1.1" 200 OK`。目前不能从该文件确认 decode 是 Python 异常退出。

### 5.2 router stdout/stderr 没有持久化到 cjy 日志

`g0033` router PID `5292`、`761528` 以及 `g0035` router PID `1136943` 的 stdout/stderr 都指向 `/dev/pts/0`：

```text
/proc/<pid>/fd/1 -> /dev/pts/0
/proc/<pid>/fd/2 -> /dev/pts/0
```

`/mnt/beegfs/cjy` 和 `/tmp` 下没有找到对应 router/gateway/smg 持久日志。因此 00:39:54 后 router 侧是否先触发 abort/断连，在当前日志面上没有证据。

### 5.3 prefill 00:40:04 的 AbortReq 更像后果，不是当前第一故障点

`sglang_cp_hicache_20260605_003303.log` 在 00:40:04 出现大量：

```text
Prefill bootstrap failed ... KVTransferError(...): Aborted by AbortReq
[OUTPUT_ZERO_DEBUG] ... finish_reason={'type': 'abort', ...}
```

这些 abort 覆盖多个 rid、多个 bootstrap room，发生在 decode0 日志停止写入之后约 10 秒。当前证据更像 decode/router/client 侧断开后 prefill transfer 被 AbortReq 取消；不能把它作为 prefill 计算/CP shared-KV 直接崩溃的根因。

### 5.4 fake warmup 的 EAGLE metadata 是单独问题，不应再当作真实 transfer 乱码证据

已读代码：

- `python/sglang/srt/disaggregation/decode.py::_is_fake_transfer()`
- `python/sglang/srt/disaggregation/fake/conn.py::FakeKVReceiver.init()`
- `python/sglang/srt/disaggregation/utils.py::MetadataBuffers`

事实：

- fake warmup 使用 `FAKE_BOOTSTRAP_HOST = "2.2.2.2"`；
- decode `_commit_transfer_to_req()` 遇到 fake transfer 时跳过 `bootstrap_room` 校验；
- `FakeKVReceiver.init()` 只设置 `has_init=True`，不会写 `MetadataBuffers`；
- EAGLE decode warmup 仍会从 metadata buffer 读取 `output_id/topk/hidden`。

因此 decode warmup 的：

```text
! of the of the of the
!!!!!!!??!??!
```

可以由未初始化/默认 metadata 解释。它是 warmup 路径 bug 或诊断噪声，但不能直接证明真实 Mooncake transfer 或 prefill CP 写入损坏。

### 5.5 当前更值得验证的真实请求风险：decode EAGLE metadata slot 生命周期

已读代码：

- `python/sglang/srt/disaggregation/decode.py::_commit_transfer_to_req()`
- `python/sglang/srt/disaggregation/decode_schedule_batch_mixin.py::process_prebuilt()`
- `python/sglang/srt/managers/scheduler_output_processor_mixin.py::_free_decode_metadata_index_if_held()`

当前逻辑：

1. `_commit_transfer_to_req()` 从 `MetadataBuffers.get_buf(idx)` 取 views；
2. EAGLE 时把 `output_topk_p/output_topk_index/output_hidden_states` view 直接挂到 `req`；
3. `process_prebuilt()` 用 `torch.as_tensor(..., device=cuda)` 和 `torch.stack(...).to(cuda)` 消费这些 CPU/pinned views；
4. `get_new_prebuilt_batch()` 的 `finally` 随后调用 `_free_decode_metadata_index_if_held()` 释放 slot。

待验证风险：

- 如果 CPU->GPU copy 在当前 stream 上是异步排队，而 metadata slot 在 copy 真正消费前被释放并被后续 transfer 覆盖，bs>1/突发请求会更容易 corrupt EAGLE topk/hidden，表现为 accept len 低和输出异常。
- `e9c341afe` 明确把原来的 clone mitigation 改成 slot ownership，commit 自身也标注 `Not-tested: Fresh ETE runtime validation...`。这是当前优先验证假设，但还没有日志直接证明。

- `is_current_only_extend_batch()`
- `should_reuse_current_extend_kv()`
- `current_extend_kv_rows_for_reuse()`
- `build_current_loc_remap()`
- `build_shared_token_kv_slot_remap()`
- `materialize_prefix_and_reuse_current_kv_page_slots()`
- `fill_current_kv_page_slots_and_remap_locs()`

当前需要验证：

- `current_extend_kv_rows_for_reuse()` 返回 `sum(extend_seq_lens_cpu)`，它不一定等于当前 attention page table 需要覆盖的 logical loc 数。
- `materialize_prefix_and_reuse_current_kv_page_slots()` 中 current loc remap 如果存在 missing logical loc，应 fail-fast 或禁用 current-only，而不能继续构造不完整 KV。

当前状态：

- 已确认：helper 层有能力表达 `prefix_pages=0` 的 current-only page-slot compose。
- 风险：`current_extend_kv_rows_for_reuse()` 是 token-count 视角；如果调用侧需要 CP-local valid rows，必须在调用侧显式传入/选择 CP-local loc。

### 4.5 padding 进入 forward 的位置

文件：`python/sglang/srt/model_executor/forward_batch_info.py`

已看函数：

- `prepare_mlp_sync_batch()`
- `_pad_inputs_to_size()`

当前理解：

- MLP sync / attn_tp / cp 对齐会把 `forward_batch.extend_num_tokens` 和张量行数 pad 到更大数，例如 40392、40448。
- CPU logical fields（`extend_seq_lens_cpu`, `seq_lens_cpu`）不应该被 padding 改写。
- 所有 cache write / index topk / MLA attention 需要明确区分：valid rows、compute rows、static-padded rows、page-padded rows。

当前状态：

- 已确认：`forward_batch.extend_num_tokens` 不能当作 request logical length。
- 待查：哪些 consumer 仍然隐式使用张量第一维或 `out_cache_loc.numel()` 作为 logical row 数。

### 4.6 disagg transfer / decode prealloc

文件：

- `python/sglang/srt/disaggregation/prefill.py`：`send_kv_chunk()`
- `python/sglang/srt/disaggregation/decode.py`：`_pre_alloc()`

已知：

- prefill transfer 的 `end_idx` 基于 `len(req.fill_ids)` / `len(req.origin_input_ids)`，再转 page indices；最后一个 partial page 仍会按 page 转移。
- decode `_pre_alloc()` 的 `fill_len` 基于 logical input/output ids，但 paged allocation 返回的 `kv_loc` 可能包含 full tail page locs。
- 需要继续检查 decode attention 是否始终用 logical `seq_lens`，不要用 `len(kv_loc)`/page padded loc 数作为真实上下文长度。

当前状态：

- 已排除：早期 decode bootstrap 失败导致的 `OUTPUT_ZERO` 不是当前 HTTP 200 乱码主线。
- 待查：decode 端 EAGLE accept 计算是否使用 logical seq len 还是 padded kv loc 数。

### 4.7 FP8 CP shared-KV persistent write

文件：

- `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`
- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py`
- `python/sglang/srt/layers/attention/nsa/quant_k_cache.py`
- `python/sglang/srt/mem_cache/memory_pool.py`
- `tai-kernel/python/tai_kernel/nsa_prefill/cp_shared_mla_store.py`
- `tai-kernel/python/tai_kernel/nsa_prefill/csrc/cp_shared_mla_store.cu`

已确认：

- 运行参数开启 `--kv-cache-dtype fp8_e4m3` 且 `SGLANG_CP_SHARED_KV_FUSED_MLA_STORE=1`。
- persistent MLA 写入优先走 `try_tai_fused_mla_store()`；失败才回到 SGLang fallback：`logical_locs -> physical_locs -> set_mla_kv_buffer()`。
- TAI FP8 fused store 对 `kv_buffer.dtype == torch.uint8` 走 fused quant store，row bytes 为 `512 FP8 + 16 scale + 128 rope bf16 = 656`。
- TAI 侧已有自洽测试，但目前看到的测试主要比较 TAI kernel 与 TAI 自己的 reference；还缺“TAI fused store 与 SGLang fallback 写同一个 raw FP8 pool 后逐字节等价”的集成单测。

高优先级待验证：

- 对相同的 CP logical locs、同样的 `k_nope/k_rope`，TAI fused store 和 SGLang fallback 写出的 `kv_buffer[layer]` 必须完全一致。
- 若该测试失败，root cause 在 persistent write；若通过，则转向 `SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1` 的 prefill->decode physical/logical 映射。

## 5. 当前最强假设（待验证）

### H1：persistent CP shared-KV 写入或 transfer 映射损坏（当前主线）

证据：

- 坏请求 `#cached-token=0`，prefill attention 可以主要依赖 current all-gather KV；但是 decode 只能消费 prefill 写入并 transfer 过来的 persistent KV。
- prefill 在同一请求中完成 `submit_write_cp_per_layer` / `final ack`，随后 decode accept len 立即掉到约 1。
- 乱码而非 shape crash，更符合“KV 内容/位置不一致”。
- FP8 下 persistent write 走 TAI fused store，尚未有与 SGLang fallback 的集成等价测试。

最小验证方向：

- 先补 CUDA 集成单测：同一批 CP logical locs 下，比较 TAI fused FP8 MLA store 与 SGLang fallback `set_mla_kv_buffer()` 的 raw buffer 字节。
- 若等价，再补 transfer 映射单测：CP rank 发送必须从 source physical locs 读，decode destination 必须按 logical page/loc 还原。

### H2：current-only MLA/index 复用的 logical length 与 current rows 覆盖不一致（降级为次线）

证据：

- 用户请求 usage prompt_tokens=40387；prefill log input=40391；HiCache logical_len=40386；forward #new-token=40448。
- 多个长度同时存在，说明当前路径同时持有 tokenizer/request length、cache logical length、page padded length、static padded length。
- 乱码 + accept len=1 比 crash 更像 KV 内容/位置不一致，而不是简单 shape mismatch。

最小验证方向：

- 在 current-only MLA/index reuse 点验证：`logical_page_table_1` 中每个会被 attention/topk 消费的 loc 都必须由 `current_locs_for_reuse` 覆盖。
- 如果未覆盖，应该 fail-fast 并打印：`seq_lens_cpu`、`extend_seq_lens_cpu`、`extend_prefix_lens_cpu`、`out_cache_loc.numel`、`page_table_1.shape/max`、`valid_current_rows`。

追加确认（2026-06-04）：

- 非 compute-padding 的 current-only index reuse 使用 all-gather 后 global `key/current_index_kv`，`_get_topk_ragged_with_cp()` 也按 per-request full valid rows 切片；这看起来是有意设计，不是当前 40k 单请求乱码的首要证据。
- MLA current-only 在 attention 前也会通过 `rebuild_cp_kv_cache()` 取得 full current KV；prefill 正常返回不能证明 persistent KV 正确。

### H3：FP8 index current TAI fallback 是性能问题，不是当前首要正确性根因

证据：

- warning 错误为 `current_index_k must be uint8`，fallback 到 Python slice/cat。
- CP1-CP7 的 `total_q_count=5056` 对该长度是正常 local valid rows。
- 仍需后续修复 dtype，让 fast path 吃到 FP8 uint8 raw view，但不应先把它当成乱码根因。

### H4：decode 端可能把 full-tail page loc 当作 logical seq 消费

证据：

- decode batch `#token=40448` 与 prefill physical page padded token 数一致。
- 这条目前只是风险，不是结论。需要继续追 `seq_lens` / page table 初始化是否用 logical fill_len。

## 6. 下一步不重复的执行清单

1. 先验证 H1：给 current-only MLA/index compose 增加窄 fail-fast 或 debug-only consistency check，优先单测复现，不直接大面积加日志。
2. 如果 H1 成立，修 source：统一当前请求的 logical/current rows 合同；不要用 silent fallback。
3. 如果 H1 不成立，再追 H3：decode transfer/prealloc 后的 `req_to_token_pool`、`seq_lens`、attention page table 是否 logical/padded 混用。
4. H2 单独排期：TAI current-index prepare dtype 应该传 uint8 raw storage，避免 FP8 下永远 fallback。

## 7. 禁止重复误区

- 不要再把 CP1-CP7 的 `total_q_count=5056` 单独当成错误；它对 40386/40391 这类长度可能是正常的。
- 不要再把 22:45-23:18 的 bootstrap 500 当成 23:28 HTTP 200 乱码的根因。
- 不要因为 `#new-token=40448` 直接判断 bug；必须定位到哪个 consumer 错把 physical padding 当 valid/logical 使用。
- 不要在没有证据前关闭 EAGLE/FP8/HiCache 作为“修复”；这只能绕过问题，不能解释输出乱码。

## 8. 当前排查边界

本 ledger 只覆盖当前 `accept len≈1 + HTTP 200 输出乱码` 主线。以下问题需要单独建账，不应混在本主线里：

- `tai_index_mqa_prepare` FP8 dtype fast-path fallback 的性能修复。
- HiCache / Mooncake transfer 带宽低、page 分散、allocator 连续性优化。
- L1/L2 free-room eviction 策略调参。
- bs>1 scheduler 组 batch 策略和 tiny extend compute-padding 性能模型。

如果后续证据证明其中某项直接导致输出乱码，再把它移动回主线并补充证据。

## 7. 2026-06-04 继续排查：新增证据

### 7.1 FP8 fused MLA persistent store 等价性单测

已新增并在远端 CUDA 环境验证：

```text
test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_fp8_fused_mla_store_matches_sglang_fallback_for_cp_owned_tail_pages
1 passed, 3 warnings in 6.08s
```

测试内容：构造 `fp8_e4m3` packed MLA cache，包含 CP owned partial tail page，比较 `try_tai_fused_mla_store()` 与 SGLang fallback `quantize_k_cache_separate + set_mla_kv_buffer_triton()` 写出的 raw uint8 buffer。

结论：在已覆盖的 CP-owned tail-page 场景下，TAI fused FP8 MLA persistent store 与 SGLang fallback 等价。后续不要把“TAI fused MLA store 本身写错 tail page”作为最高优先级假设，除非出现更强反证。

### 7.2 prefill 与 decode 实际运行代码路径不一致

已用 `sha256sum` 比较本地、g0034 prefill、g0035/g0036 decode 容器内关键文件：

- 本地 `/root/sglang-work/sglang-dev` 与 `g0034:/sgl-workspace/sglang-tai` 完全一致。
- `g0035/g0036:/sgl-workspace/sglang-tai` 也与本地一致。
- 但 decode 日志栈路径是 `/sgl-workspace/sglang/...`，而 `g0035/g0036:/sgl-workspace/sglang` 的关键文件 hash 与本地不同：
  - `nsa_backend.py`
  - `nsa_indexer.py`
  - `utils.py`
  - `memory_pool.py`
  - `disaggregation/mooncake/conn.py`

证据摘录：

```text
local nsa_backend.py: 6664f004024b8849...
g0034 /sgl-workspace/sglang-tai nsa_backend.py: 6664f004024b8849...
g0035 /sgl-workspace/sglang nsa_backend.py: 197624526cace147...
g0036 /sgl-workspace/sglang nsa_backend.py: 197624526cace147...
```

结论：当前线上 prefill 和 decode 很可能不是同一份 CP shared-KV/FP8/page-tail 合同。由于本次乱码发生在 prefill HTTP 200 后、decode 消费 transferred KV 时，代码版本不一致现在是最高优先级根因候选。

后续动作：

1. 先不要继续在 prefill 侧反复排查已验证的 store 路径。
2. 需要确认 decode 实际 bind mount 与启动目录，确保 decode 使用与 prefill 相同的代码。
3. 修复/同步后需要重启 decode 才能验证 accept len/output 是否恢复；仅 scp 文件不会影响已运行 Python 进程。

### 7.3 decode 容器挂载已确认：`/sgl-workspace/sglang` 是镜像内旧代码

`docker inspect sglang-glm5-dev-2`：

```text
g0035: /mnt/beegfs/cjy/sglang-dev -> /sgl-workspace/sglang-tai
g0036: /mnt/beegfs/cjy/sglang-dev -> /sgl-workspace/sglang-tai
```

没有 `/sgl-workspace/sglang` 的 bind mount。但 decode 栈路径是：

```text
/sgl-workspace/sglang/python/sglang/...
```

结论：decode 当前实际从容器镜像内旧代码启动；`/mnt/beegfs/cjy/sglang-dev` 的同步不会影响当前 decode 启动目录。prefill 与 decode 的 CP shared-KV/FP8/page-tail/transfer 合同不一致不是理论风险，而是当前运行事实。

正确修复方向：decode 启动脚本/工作目录必须切到 `/sgl-workspace/sglang-tai`，或在重启前显式更新 `/sgl-workspace/sglang`。仅修改 prefill 或 cjy host path 不能修复当前 accept len=1/输出乱码。

### 7.4 已加入代码级防线：prefill/decode runtime source fingerprint fail-fast

变更：`python/sglang/srt/disaggregation/common/conn.py`

- bootstrap parallel info 新增：
  - `runtime_source_fingerprint`
  - `runtime_source_root`
- fingerprint 覆盖当前 CP shared-KV / FP8 / page-tail 合同的关键源码：
  - `sglang.srt.layers.attention.nsa_backend`
  - `sglang.srt.layers.attention.nsa.nsa_indexer`
  - `sglang.srt.layers.attention.nsa.utils`
  - `sglang.srt.mem_cache.memory_pool`
  - `sglang.srt.disaggregation.mooncake.conn`
- decode 拉取 prefill parallel info 时，如果 prefill fingerprint 与 decode 本地 fingerprint 不一致，直接 `RuntimeError`，避免继续运行到 silent KV corruption / accept len=1 / 输出乱码。

新增单测：

```text
test/registered/unit/disaggregation/test_common_conn_runtime_fingerprint.py
```

RED：远端初跑 2 failed，原因是 `common.conn` 尚无 `get_runtime_source_fingerprint`。

GREEN：实现后远端验证：

```text
PYTHONPATH=python python -m pytest -q \
  test/registered/unit/disaggregation/test_common_conn_runtime_fingerprint.py \
  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_fp8_fused_mla_store_matches_sglang_fallback_for_cp_owned_tail_pages

3 passed, 3 warnings in 6.13s
```

注意：这个防线会阻止后续同类错配静默运行；它不会改变已经启动的旧 decode 进程。当前远端要恢复输出质量，仍需要让 decode 进程从 `/sgl-workspace/sglang-tai` 或已同步的同源代码重启。

### 7.5 相关 transfer mapping 回归验证

远端验证：

```text
PYTHONPATH=python python -m pytest -q \
  test/registered/unit/disaggregation/test_common_conn_runtime_fingerprint.py \
  test/registered/unit/disaggregation/test_cp_shared_kv_transfer_mapping.py \
  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_fp8_fused_mla_store_matches_sglang_fallback_for_cp_owned_tail_pages

10 passed, 3 warnings in 6.00s
```

覆盖范围：

- runtime source fingerprint mismatch fail-fast；
- CP shared-KV transfer 的 logical page -> physical source page / logical destination position 映射；
- valid tail page 不按 cp_size 过度 padding；
- FP8 fused MLA persistent store 与 fallback 等价。

### 7.6 fingerprint 实现修正：避免 import 副作用

实现细节已修正：fingerprint 不再对每个目标模块调用 `importlib.find_spec(module)`，避免触发 `mooncake.__init__` 这类父包 import。现在只对已加载的 `sglang` 包定位 `python/` 根目录，再按模块名拼出源码路径读取 bytes。

远端实际检查：

```text
get_runtime_source_root() -> /sgl-workspace/sglang-tai
len(get_runtime_source_fingerprint()) -> 64
fingerprint prefix -> 0c0e736fa6de304f
```

回归：同 7.5 的 10 条相关测试继续通过。

### 7.7 g0035/g0036 挂载代码已具备 fingerprint 防线，但当前 decode 启动目录仍旧

远端检查：

```text
g0035 /sgl-workspace/sglang-tai fingerprint: 0c0e736fa6de304f
g0036 /sgl-workspace/sglang-tai fingerprint: 0c0e736fa6de304f
```

同时 `/sgl-workspace/sglang/python/sglang/srt/disaggregation/common/conn.py` 中没有 `get_runtime_source_fingerprint`，说明当前 decode 日志路径对应的旧目录仍不具备新防线。

结论：下一轮 ETE 必须确保 decode 启动工作目录/PYTHONPATH 指向 `/sgl-workspace/sglang-tai`。如果仍从 `/sgl-workspace/sglang` 启动，既不会使用最新修复，也不会触发 fingerprint fail-fast。

### 7.8 当前 decode 主进程 cwd 已确认

通过容器内 `/proc/<pid>/cwd`：

```text
g0035 decode PID=15207 cwd=/sgl-workspace/sglang
g0036 decode PID=14525 cwd=/sgl-workspace/sglang
```

这与日志栈路径一致，排除“只是日志引用旧路径”的可能。当前 accept len=1 / 乱码验证环境确实是 prefill 新代码 + decode 旧镜像代码混跑。

### 7.9 纠正：cwd 不是根因判据，必须以 Python 实际 import source 为准

用户指出：`python -m sglang.launch_server` 从 `/sgl-workspace/sglang` 启动并不必然有问题；如果安装位置或 `PYTHONPATH` 指向正确代码，cwd 可以无影响。

修正：前文 7.3/7.8 中“cwd=/sgl-workspace/sglang 因此一定旧代码”的结论过强。当前可作为证据的是：

- traceback/code object path 显示 decode 运行栈来自 `/sgl-workspace/sglang/python/sglang/...`；
- `/sgl-workspace/sglang` 与 `/sgl-workspace/sglang-tai` 的关键文件 hash 不一致；
- 但仍需确认运行进程 `PYTHONPATH`、editable install、`sglang.__file__` 的实际来源。

后续排查必须以 import source/安装位置为准，不再单独用 cwd 下结论。

### 7.10 再次修正：当前 Python install source 实际一致

检查方式：在 g0034/g0035/g0036 容器内分别从 `/sgl-workspace/sglang-tai` 和 `/sgl-workspace/sglang` 执行：

```python
import sglang
import sglang.srt.layers.attention.nsa_backend as nb
from sglang.srt.disaggregation.common.conn import get_runtime_source_fingerprint, get_runtime_source_root
```

结果均为：

```text
sglang.__file__ = /sgl-workspace/sglang-tai/python/sglang/__init__.py
nsa_backend.__file__ = /sgl-workspace/sglang-tai/python/sglang/srt/layers/attention/nsa_backend.py
source_root = /sgl-workspace/sglang-tai
fingerprint prefix = 0c0e736fa6de304f
```

结论：当前环境下 cwd 不影响 `python -m` 的 import source；安装位置/PYTHONPATH 已指向 `/sgl-workspace/sglang-tai`。7.2/7.3/7.8 中将 `/sgl-workspace/sglang` 目录 hash 差异作为根因的判断不成立，只能保留 fingerprint fail-fast 作为未来防线。

当前根因需要回到 CP shared-KV / FP8 / bs>1 / page-tail / decode transfer 消费路径本身继续排查。

## 2026-06-04 note: why decode can regress even when CP is disabled

The current decode process does not enable CP shared-KV (`SGLANG_CP_DRAFT_SHARED_KV` is absent and decode server args show `enable_nsa_prefill_context_parallel=False`, `enable_nsa_prefill_cp_shared_kv=False`). That does **not** isolate it from recent CP-driven changes because several touched files are shared by normal decode:

- `speculative/eagle_worker.py` runs for any EAGLE decode. The recent `cp_local_hidden_states` marker is derived from `logits_output.draft_hidden_states is not None`, while the actual CP draft path is gated by `_can_use_cp_draft_shared_kv(batch)`. If `draft_hidden_states` is populated outside CP, decode may incorrectly treat hidden states as CP-local and skip static padding.
- `model_executor/forward_batch_info.py::_pad_inputs_to_size()` is generic static-padding code used by EAGLE/DP/MLP-sync, not a CP-only path.
- `models/deepseek_v2.py` always has the hidden-state capture hook (`capture_draft_hidden_states`) and can populate `logits_output.draft_hidden_states` when requested by EAGLE worker.
- Decode also consumes prefill-produced KV through disaggregation transfer, so prefill CP corruption can show up in decode even if decode itself is not CP-enabled. However, the latest evidence shows decode warmup itself already emits abnormal text, so the current primary suspicion is decode-side EAGLE/FP8/static-padding behavior rather than only prefill HiCache.

Current hypothesis to test next: `cp_local_draft_hidden_states` should be true only when `_can_use_cp_draft_shared_kv(batch)` is true, not merely when `logits_output.draft_hidden_states` exists.

## 2026-06-04 prefill KV corruption lead: current reuse uses global locs in CP path

New primary lead after re-centering on prefill-produced KV:

- Decode may not enable CP, but disagg decode consumes prefill-produced KV. Bad prefill CP shared-KV writes/composes can directly cause low EAGLE accept length and garbage decode text.
- In `nsa_backend.py`, MLA partial/current reuse has two branches:
  - compute-padding branch uses `select_cp_local_valid_rows_for_cache_write()` and `get_cp_shared_kv_local_out_cache_loc()`.
  - non-compute-padding branch still uses `k[:valid_current_rows]`, `k_rope[:valid_current_rows]`, and `forward_batch.out_cache_loc[:valid_current_rows]`.
- In CP shared-KV, the local current rows and local locs must come from the same CP split contract. Using the global `out_cache_loc` prefix can mismatch this rank's local rows and produce a bad page-slot compose.
- Existing unit test explicitly expected the old global loc prefix; changed it to RED against this bug.
- Separate code defect found: `_cp_split_and_rebuild_batch_in_seq()` passes `input_` twice to `split_tensor_by_cp_batch_plan`, inconsistent with the current signature. This should be fixed in the same CP split consistency pass.

## 2026-06-04 fix applied: CP-local current reuse for MLA and index

Applied local fixes for the prefill-produced KV corruption hypothesis:

- `nsa_backend.py` MLA partial/current reuse no longer uses `forward_batch.out_cache_loc[:valid_current_rows]` in the non-compute-padding branch. It now obtains `get_cp_shared_kv_local_out_cache_loc()` and ensures current KV rows are CP-local, using `cp_split_and_rebuild_data()` when the source tensors are still gathered/global.
- `nsa_indexer.py` index current reuse now follows the same contract: local locs + local key rows. The old non-compute-padding path used global `out_cache_loc` prefix and global `key[:valid_current_rows]`.
- `cp_split_and_rebuild_data()` now preserves all non-token dimensions (`view(-1, *shape[1:])`) so MLA/index KV tensors do not collapse from 3D to 2D when CP-split.

Verification so far:

- RED observed for MLA source-gate test before code change.
- RED observed for index source-gate test before code change.
- Local py_compile passed for touched code.
- Local targeted pytest passed for both source-gate tests.
- Remote py_compile and targeted pytest pending after sync.

Remote verification after sync:

- `python -m py_compile` passed for `nsa_indexer.py`, `nsa_backend.py`, `nsa/utils.py`, and the runtime test file in `g0034` container.
- Remote targeted pytest passed:
  - `test_index_current_reuse_prepare_accepts_padded_out_cache_loc`
  - `test_mla_current_reuse_gate_accepts_padded_out_cache_loc`
  - `test_fp8_fused_mla_store_matches_sglang_fallback_for_cp_owned_tail_pages`
- Remote minimal script verified `cp_split_and_rebuild_data()` preserves 3D shape `(tokens, heads, dim)`.
- Full `test_nsa_cp_utils.py` import is currently blocked by remote `sgl_kernel` missing `gelu_tanh_and_mul`; keep using targeted/minimal verification until the extension install is corrected.

Additional same-class fix:

- `_maybe_materialize_shared_index_buffer()` also had a non-compute-padding partial-current compose branch using global `forward_batch.out_cache_loc` and slicing `current_index_kv` by global rows. After `forward_cuda()` now prepares CP-local `current_index_kv`, this compose function must consume CP-local locs as well.
- Added a stricter source-gate test anchored inside `_maybe_materialize_shared_index_buffer()`; observed RED before the fix and GREEN after the fix.

## 2026-06-04 full prefill-flow audit pass: confirmed flow boundaries

Checked the end-to-end prefill path again from scheduler to decode handoff.

Code evidence:

- `disaggregation/prefill.py::event_loop_normal_disagg_prefill()` runs `recv -> bootstrap_queue -> get_new_batch_prefill/process_chunk -> run_batch -> process_batch_result -> process_inflight_queue`.
- `process_batch_result_disagg_prefill()` appends the first target token, inserts/locks radix via `tree_cache.cache_unfinished_req(req)`, queues KV transfer, and copies EAGLE handoff tensors from `batch.spec_info` into the request before `send_kv_chunk(last_chunk=True)`.
- `send_kv_chunk()` transfers page ids derived from `req_to_token_pool.req_to_token[start:end]`; non-last chunks floor to page boundary, last chunks use `len(origin_input_ids)`.
- `CPSharedPagedTokenToKVPoolAllocator` explicitly stores **logical** page ids in scheduler/radix/`req_to_token_pool`; physical translation is deferred to `CpSharedKVLayout`. Therefore the Mooncake `filter_kv_pages_for_cp_shared_kv()` path is internally consistent **if** no physical page loc leaks into `req_to_token_pool`.

Remote evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260604_003716.log`:

- Prefill was launched with `enable_cp_shared_kv_prefill_bs_gt1=True`, `kv_cache_dtype='fp8_e4m3'`, `hicache_io_backend='direct'`, `hicache_mem_layout='page_first_direct'`, `chunked_prefill_size=65536`.
- For the 40k prompt, prefill logged `logical_len=40386` but `#new-token: 40448`, matching page-rounded compute/cache coverage.
- The hot path still emits repeated `[CP_SHARED_KV_FALLBACK][tai_index_mqa_prepare] ... error=current_index_k must be uint8`, so current-index TAI fast path is not being used.

Conclusion: the transfer page filtering path is not the first suspect by itself. The stronger correctness suspects remain the CP shared-KV current/index materialization and the EAGLE handoff tensors produced by prefill.

## 2026-06-04 high-risk finding: EAGLE prefill handoff reads CP-local hidden as if it were batch-major

This is a separate high-risk accept-length lead from KV/index corruption.

Code evidence:

- `speculative/eagle_worker.py::forward_target_extend()` sets `model_worker_batch.capture_draft_hidden_states = True` when CP draft shared-KV is enabled. The target model then captures `logits_output.draft_hidden_states` before CP output collection.
- `models/deepseek_v2.py` captures `forward_batch.draft_hidden_states = hidden_states` at line-level before `cp_collect_last_token_hidden()` / `cp_all_gather_rerange_output()`. Under NSA prefill CP, that tensor is CP-local token-major, not batch-major last-token hidden.
- `disaggregation/prefill.py::process_batch_result_disagg_prefill()` currently stores `req.hidden_states_tensor = batch.spec_info.hidden_states[i].cpu().clone()`. That assumes `batch.spec_info.hidden_states` is `[batch, hidden]`.
- In the CP-local draft path, `batch.spec_info.hidden_states` can be `[local_tokens, hidden]`. For `bs=1`, indexing `[0]` stores the first local token hidden, not the request's last-token hidden. For `bs>1`, this is even less likely to match the correct per-request handoff row.
- Decode-side `decode_schedule_batch_mixin.py` stacks `req.hidden_states_tensor` into `EagleDraftInput(hidden_states=[bs, hidden])`, so the wrong prefill row becomes the draft model side-channel.

Impact:

- This can directly explain low EAGLE accept length, because the draft model receives the wrong prefill hidden state.
- It does **not** by itself prove target KV corruption; target verification should reject bad draft tokens. Therefore it explains accept-len collapse more strongly than garbage target text.

Required fix direction:

- Prefill EAGLE handoff must explicitly collect the per-request last-token hidden from the CP-local draft hidden tensor before copying it to `req.hidden_states_tensor`.
- It should use the same owner/local-offset contract as `cp_collect_last_token_hidden()` / batch metadata, not `hidden_states[i]`.
- Add a unit test where `hidden_states` is token-major CP-local and `bs=1/bs>1`; verify the stored hidden row equals the true request last token, not row 0.

## 2026-06-04 high-risk finding: current-only index top-k uses global request offsets on CP-local current_index_kv

Code evidence:

- `nsa_indexer.py::forward_cuda()` now prepares `current_index_kv` from CP-local current rows using `get_cp_shared_kv_local_out_cache_loc()` and either local `key` rows or `cp_split_and_rebuild_data()`.
- `_get_topk_in_seq_cp_pair_batch()` passes `current_index_kv` directly to `_get_topk_ragged_with_cp()` when `current_only_batch=True`.
- `_get_topk_ragged_with_cp()` builds `current_req_offsets` by accumulating global `forward_batch.extend_seq_lens_cpu`.
- It then passes these global offsets as `current_bases` to the TAI current-index prepare path and uses the same offsets in the Python fallback slice.

Why this is wrong:

- After the recent CP-local current-index fix, `current_index_kv` is no longer global flattened request rows. It is local flattened rows in CP split order.
- For `bs>1`, request 1+ local base is not the sum of previous global extend lengths. It must come from batch-plan CP-local request offsets (`request_rank_local_offsets` / valid split metadata), and it must account for compute padding only where the local source tensor includes compute-padded rows.

Additional concrete code bug:

- In the Python fallback loop, the check `if pre_chunk_offset != 0` references `pre_chunk_offset` from the previous segment-building loop, not the current `segment_record`. This can test the wrong request/segment and is unsafe even before performance tuning.

Impact:

- This can corrupt index top-k for current-only cache-miss/prefill bs>1 paths.
- Remote fallback logs show this path is active: `[CP_SHARED_KV_FALLBACK][tai_index_mqa_prepare] reason=current_batch_kernel_failed ... segments=2 total_kv_len≈43k total_q_count=5056`.
- This is currently one of the strongest candidates for bad prefill-produced KV/index state under bs>1.

Required fix direction:

- Build a per-request CP-local current row base vector from `CPSharedKVBatchPlan` and pass that to both TAI and Python current-index top-k paths.
- Include `pre_chunk_offset` in `segment_records` instead of reading the loop variable after the fact.
- Add a unit test with two requests whose global extend lengths differ and whose CP-local row counts differ; assert `_get_topk_ragged_with_cp()` slices from the local base, not global base.

## 2026-06-04 performance/correctness finding: current-index TAI fallback is caused by dtype contract mismatch

Code evidence:

- `tai-kernel/python/tai_kernel/nsa_prefill/cp_index_mqa_prepare.py::_validate_current_batch_inputs()` requires `current_index_k.dtype == torch.uint8`.
- `nsa_indexer.py::forward_cuda()` stores `current_k_fp8` as a `torch.float8_e4m3fn` tensor in `current_index_kv`.
- `cp_shared_kv_runtime.py::try_tai_prepare_cp_mqa_current_index_batch()` passes `current_index_k` through directly, so the TAI path rejects it with `current_index_k must be uint8`.

Remote evidence:

- Latest prefill logs contain repeated `current_index_k must be uint8` warnings across ranks and layers.

Impact:

- This is definitely a hot-path performance regression.
- Because the fallback path currently has the local/global offset issue above, this dtype mismatch also increases exposure to the buggy Python fallback.

Required fix direction:

- Either pass `current_index_k.view(torch.uint8)` into the TAI wrapper, or update the TAI wrapper to accept float8 and reinterpret internally.
- Keep the Python fallback as fail-fast/diagnostic for CUDA hot path until local-base correctness is fixed.

## 2026-06-04 HiCache/chunked prefill stale-tail status

Code evidence:

- `hiradix_cache.py::_build_prepare_write_backup_candidate_for_req()` floors chunked-prefill backup end to a completed page and skips backup when no full page is available.
- `_cp_prune_stale_tail_after_page_floor()` still raises `HiCachePendingBackupSplit` if a stale tail subtree has lock refs, host refs, or pending host write state. It drains finished write acks before declaring blocked, but it still fails if the state remains unprunable.
- `match_prefix()` catches `HiCachePendingBackupSplit` and returns a deferred node in match results, but insertion/cache-unfinished paths can still surface this if chunked insertion attempts to prune while backup remains pending.

Impact:

- The previous chunked-prefill crash is mitigated for some cases by page-floor backup, but not eliminated by construction.
- Risk remains when chunked prefill extends a node whose stale tail is still protected or backup-pending.

Required fix direction:

- Insert/cache-unfinished should defer and retry this request, not crash the scheduler, when stale-tail prune is blocked by in-flight backup.
- A regression test should simulate: first chunk creates sub-page valid tail, async backup pending, next chunk extends past page boundary; expected result is deferred insertion/rollback, not uncaught `HiCachePendingBackupSplit`.

## 2026-06-04 prefetch timing audit status

Code evidence:

- `nsa_backend.py::_maybe_start_cp_shared_kv_attention_prefetch()` is now called immediately before the current layer attention kernel.
- Both index and MLA prefetchers materialize the next-layer prefix on the current stream first, then enqueue the CP all-reduce on a shared prefetch stream.
- `wait_attention_window()` is deliberately deferred; the real wait happens when the next layer consumes the prefetched handle. This matches the intended "next layer consume waits" model.
- Draft/EAGLE inputs do not create MLA/index prefetchers (`disable_draft_prefetch = cp_shared_kv_is_draft_input(forward_batch)`).

Risks:

- The prefix materialize step still allocates and fills a dense buffer before the async all-reduce. For large prefixes this is non-trivial CPU/GPU launch overhead and can still contend with current-layer work if the hook is placed too early or if the dense buffer allocation is expensive.
- The async collective is still an all-reduce over dense materialized prefix rows/pages. Even when overlapped, it consumes communication resources and can interfere with other CP/TP collectives.
- Because wait is deferred, a bad handle/key mismatch should fail at consume. Missing consume-side fail-fast coverage remains important.

Current conclusion:

- Prefetch timing is not the strongest correctness suspect for garbage output in the latest logs. It remains a performance and hang-risk area, especially under long prefixes and bs>1, but the latest concrete correctness evidence points more directly at EAGLE handoff and current-index top-k local/global offsets.

## 2026-06-04 P0/P1 repair plan

Scope for this repair pass:

1. P0 EAGLE prefill handoff correctness
   - Add a small helper that converts CP-local draft hidden states to the per-request hidden rows expected by disaggregated decode.
   - Use batch-plan last-token owner/local-offset metadata when available; otherwise reuse the single-request in-seq last-token owner calculation.
   - Keep non-CP/full hidden behavior unchanged.
   - Fail fast if CP-local hidden is marked but required CP metadata is missing or points outside the local hidden tensor.

2. P0 current-only index top-k bs>1 local base correctness
   - Stop deriving `current_req_offsets` from global `extend_seq_lens_cpu` when `current_index_kv` is CP-local.
   - Derive per-request local current bases from `CPSharedKVBatchPlan` rank-local valid metadata.
   - Store `pre_chunk_offset` in each segment record so fallback checks the segment it is actually processing.

3. P1 TAI current-index dtype contract
   - Pass the current-index K tensor to tai-kernel as uint8 bytes, matching `tai_kernel.nsa_prefill.cp_index_mqa_prepare` validation.
   - Keep the returned buffer interpreted as fp8 at the callsite.

Tests to add before production changes:

- EAGLE CP-local handoff selects the true last-token local row for bs=1 and bs>1, not `hidden_states[i]`.
- Current-index top-k local base builder returns CP-local bases for bs>1 and does not use global extend offsets.
- TAI wrapper receives uint8 current-index K for fp8 input.

## 2026-06-04 P0 refinement: EAGLE hidden handoff marker must be reset after draft forward

Additional code-flow evidence:

- Target `DeepseekV3ForCausalLM.forward_core()` stores `forward_batch.draft_hidden_states` before CP output collect so the draft model can consume CP-local target hidden rows.
- `EagleWorker.forward_draft_extend()` marks the temporary `EagleDraftInput.hidden_states` as `cp_local_hidden_states=True` while the draft model consumes that CP-local side channel.
- `DeepseekModelNextN.forward()` then collects the draft model output hidden via `cp_collect_last_token_hidden()` when using CP-local draft input, so `logits_output.hidden_states` from the draft forward is expected to be batch-major `[bs, hidden]`.
- `EagleWorker.capture_for_decode()` overwrites `draft_input.hidden_states` with this draft output, but currently leaves `draft_input.cp_local_hidden_states=True`.

Correction to the earlier suspect:

- The prefill handoff does not directly copy the original target CP-local hidden side channel after draft forward; it copies `batch.spec_info.hidden_states` after `capture_for_decode()`.
- The remaining P0 risk is the stale semantic marker: after `capture_for_decode()`, the tensor is no longer CP-local, so leaving `cp_local_hidden_states=True` can make later static-padding / draft-input consumers treat a batch-major hidden tensor as CP-local.

Required fix direction:

- Reset `EagleDraftInput.cp_local_hidden_states=False` immediately after `capture_for_decode()` replaces `hidden_states` with draft output hidden.
- Add a unit test that starts with `cp_local_hidden_states=True`, calls `capture_for_decode()`, and verifies the hidden tensor is replaced by draft output while the CP-local marker is cleared.

## 2026-06-04 P0/P1 implementation status

Implemented in this pass:

1. EAGLE draft hidden marker cleanup
   - `EagleWorker.capture_for_decode()` now clears `EagleDraftInput.cp_local_hidden_states` after replacing the temporary target CP-local side-channel with the draft model output hidden.
   - This keeps the contract precise: CP-local marker is only valid before/during draft model consumption, not after decode handoff hidden is batch-major.

2. Current-index top-k CP-local bases
   - `_get_topk_ragged_with_cp()` now derives current-index request bases via `_build_current_index_request_bases()`.
   - Legacy/no-plan current tensors still use global extend offsets.
   - CP shared-KV batch-plan current tensors use `request_valid_rank_local_offsets` first, then `request_rank_local_offsets`, because the current-index K/S buffers are flattened CP-rank-local valid rows.
   - Missing local bases under a batch plan fail fast instead of silently using global offsets.

3. TAI current-index dtype ABI
   - The TAI current-index batch prepare path now passes fp8 K tensors as `uint8` views via `_current_index_k_for_tai()`.
   - Existing uint8 callers are unchanged; unknown dtypes still pass through to the wrapper, which keeps its own validation/fallback behavior.

Regression coverage added:

- `test_indexer_ragged_cp_index_current_batch_uses_cp_local_bases_and_uint8_k`
  - RED before fix: TAI received `torch.float8_e4m3fn` and global bases `[0, 5]`.
  - GREEN after fix: TAI receives `torch.uint8` and CP-local bases `[0, 2]`.
- `test_eagle_capture_for_decode_clears_cp_local_hidden_marker`
  - RED before fix: marker stayed `True` after capture.
  - GREEN after fix: marker is cleared while hidden points to the draft output tensor.

Verification:

- Remote container targeted RED was observed before production changes: both new tests failed for the expected reasons.
- Remote container after fix:
  - `python -m py_compile python/sglang/srt/layers/attention/nsa/nsa_indexer.py python/sglang/srt/speculative/eagle_worker.py test/registered/unit/layers/test_nsa_cp_utils.py`
  - targeted current-index/EAGLE tests: `2 passed`
  - current-index related subset: `6 passed`
  - full `test/registered/unit/layers/test_nsa_cp_utils.py`: `87 passed`

Remaining runtime risk:

- This fixes two concrete CP shared-KV correctness hazards, but it does not prove the remote ETE garbage-output issue is fully resolved. The next ETE run should specifically check that the `current_index_k must be uint8` fallback warning disappears and that EAGLE accept length recovers above the previous ~1-2 range.

Post-cleanup verification note:

- `_current_index_k_for_tai()` was made robust to PyTorch builds that may not expose every fp8 dtype symbol.
- Re-ran remote full `test/registered/unit/layers/test_nsa_cp_utils.py`: `87 passed`.

## 2026-06-04 full prefill-flow audit pass

Scope:

- Re-read the current prefill inference path after `f50e2b1e0` from scheduler admission through target forward, CP metadata, NSA index/MLA current reuse, EAGLE draft handoff, HiCache backup/insert, and Mooncake disaggregation transfer.
- This pass is an audit/ledger update only. No code was changed in this pass.

Checked path and direct evidence:

1. Scheduler/disaggregated prefill main loop
   - `python/sglang/srt/disaggregation/prefill.py:529-551` receives bootstrapped requests, selects a prefill batch, runs `run_batch()`, then calls `process_batch_result()`.
   - `python/sglang/srt/managers/schedule_batch.py:1481-1537` constructs logical extend inputs and calls `alloc_for_extend()` before model forward.
   - `python/sglang/srt/managers/scheduler.py:2635-2703` registers CP HiCache write backups after KV slot allocation and immediately before model forward.
   - `python/sglang/srt/disaggregation/prefill.py:635-668` updates radix/HiCache, stores EAGLE top-k/hidden side channel, then sends KV chunks.

2. Target model CP metadata and hidden contract
   - `python/sglang/srt/models/deepseek_v2.py:2237-2250` creates `forward_batch.nsa_cp_metadata` only when `can_cp_split(...)` succeeds.
   - `python/sglang/srt/models/deepseek_v2.py:2076-2088` captures draft hidden before final target CP output collect, then target output is either narrow last-token collect or full gather/rerange.
   - Inference: CP-local target hidden is a temporary side channel for draft model consumption; prefill-to-decode hidden should be batch-major after draft forward, not CP-local.

3. EAGLE/draft handoff
   - `python/sglang/srt/speculative/eagle_worker.py:284-308` enables CP draft shared KV from env/model-runner capability and extend mode; it does not explicitly check that current `forward_batch.nsa_cp_metadata` exists.
   - `python/sglang/srt/speculative/eagle_worker.py:332-354` passes `logits_output.draft_hidden_states` as CP-local draft side channel when present.
   - `python/sglang/srt/speculative/eagle_worker.py:1069-1075` now clears `cp_local_hidden_states` after `capture_for_decode()` replaces hidden with draft output.
   - `python/sglang/srt/models/deepseek_nextn.py:195-307` still has a bs=1 compatibility branch that pads CP-local hidden by up to `cp_size` rows; bs>1 fail-fasts.

4. CP split / compute padding / direct-write cache loc contract
   - `python/sglang/srt/layers/attention/nsa/utils.py:500-755` builds request-first batch CP split metadata. Valid rows and compute-padded rows are tracked separately; compute pages are `max(valid_pages, cp_size)`, not `2*cp_size`.
   - `python/sglang/srt/layers/attention/nsa/utils.py:783-1005` supports `split_kind="compute"` vs `split_kind="valid"`; cache writes must use valid rows.
   - `python/sglang/srt/layers/attention/nsa/utils.py:1634-1805` direct-write local `out_cache_loc` uses valid split rows and slices only allowed static padding.
   - `python/sglang/srt/mem_cache/cp_shared_kv_layout.py:56-101` preserves dummy page 0 and maps usable logical pages from page 1 by CP owner.
   - Conclusion: current direct-write/page0/valid-vs-compute contract is internally consistent and is not the strongest current correctness suspect.

5. NSA index/MLA current and partial-current reuse
   - `python/sglang/srt/layers/attention/nsa/nsa_indexer.py:136-153` converts fp8 current-index K to `uint8` for the TAI ABI.
   - `python/sglang/srt/layers/attention/nsa/nsa_indexer.py:155-192` builds request bases from CP-local batch-plan offsets when present.
   - `python/sglang/srt/layers/attention/nsa/nsa_indexer.py:1916-2135` handles current-only and partial-current top-k paths, with current-index KV expected to be CP-local under batch plan.
   - `python/sglang/srt/layers/attention/nsa_backend.py:1802-2125` selects valid current rows for compute padding, supports current-only page-slot compose, and falls back to synchronous partial-current compose when prefetch is absent.
   - `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py:2215-2305` batch prefix slot span may cover a contiguous flattened slot range that includes non-prefix/current slots between requests; those are later overwritten/masked.
   - `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py:1855-2035` TAI IPC materialize fast path only supports `start_slot == 0`; nonzero spans fall back to local materialize/collective path.

6. Prefetch
   - `python/sglang/srt/layers/attention/nsa/cp_shared_kv_prefetch.py:334-450` MLA prefetcher creation currently supports only `batch_size == 1` and skips short prefixes/extends.
   - `python/sglang/srt/layers/attention/nsa/cp_shared_kv_prefetch.py:790-963` materializes the next-layer prefix on the current stream first, then enqueues async all-reduce on the prefetch stream.
   - `python/sglang/srt/layers/attention/nsa/cp_shared_kv_prefetch.py:1023-1030` attention-window wait is deliberately deferred; consume waits later.
   - `python/sglang/srt/layers/attention/nsa/cp_shared_kv_prefetch.py:1086-1155` index prefetcher also rejects bs>1.
   - Conclusion: bs>1 currently loses async L2->L1 prefetch fast path. That is a performance risk more than the strongest garbage-output suspect.

7. L1 owner-lane allocator
   - `python/sglang/srt/mem_cache/common.py:447-633` requires CP owner page planning for CP shared-KV in-seq split; allocation retries after owner-lane eviction and raises `KVCapacityWaitError` on lane exhaustion.
   - `python/sglang/srt/mem_cache/allocator.py:930-1045` allocates pages by CP owner lane and returns full logical `out_cache_loc` in request order.
   - `python/sglang/srt/mem_cache/allocator.py:1047-1101` HiCache reload can allocate pages with saved owner pattern.
   - Risk is mainly fragmentation/performance: owner buckets are consumed by sorted prefix selection, not by a best contiguous extent search.

8. HiCache/radix/chunked backup
   - `python/sglang/srt/mem_cache/hiradix_cache.py:2365-2388` stale-tail prune raises `HiCachePendingBackupSplit` if the stale subtree is still locked/host-referenced/write-pending.
   - `python/sglang/srt/mem_cache/hiradix_cache.py:2444-2510` chunked CP backup is floored to completed pages and skipped when no full page exists.
   - `python/sglang/srt/mem_cache/hiradix_cache.py:4068-4209` `insert()` pre-checks pending split state before `_split_node()`, but still calls `_cp_prune_stale_tail_after_page_floor()` after splitting.
   - `python/sglang/srt/mem_cache/radix_cache.py:624-675` `cache_unfinished_req()` rolls back prepared backup if `insert()` returns `pending_backup_deferred_node`.
   - Remaining risk: post-split stale-tail prune can still raise if the stale tail becomes/continues blocked after the pre-check. Previous chunked crash class is mitigated but not eliminated by construction.

9. Disaggregation/Mooncake prefill-to-decode transfer
   - `python/sglang/srt/disaggregation/prefill.py:870-1005` floors non-last chunk to page boundary, gathers page indices, and sends KV chunks.
   - `python/sglang/srt/disaggregation/utils.py:633-651` filters pages by CP owner and returns absolute request page positions.
   - `python/sglang/srt/disaggregation/mooncake/conn.py:1414-1453` sender replaces raw page slice with CP-owned logical pages plus logical positions.
   - `python/sglang/srt/disaggregation/mooncake/conn.py:1023-1031` receiver maps those logical positions into decode destination pages.
   - `python/sglang/srt/disaggregation/mooncake/conn.py:1055-1065` still truncates prefill pages if destination pages are shorter. The comment says this should never happen.
   - `python/sglang/srt/disaggregation/mooncake/conn.py:338-450` transfer bandwidth depends on contiguous source/destination groups and whether layer transfers are batched.
   - `python/sglang/srt/disaggregation/common/conn.py:50-56` runtime fingerprint covers only a narrow module subset and does not include `cp_shared_kv_runtime.py`, `cp_shared_kv_prefetch.py`, `eagle_worker.py`, `deepseek_nextn.py`, or tai-kernel version.

Ranked remaining risks after this pass:

1. Highest correctness risk: Mooncake CP shared-KV transfer mismatch is truncated instead of fail-fast.
   - Evidence: `mooncake/conn.py:1055-1065` silently shortens `kv_chunk.prefill_kv_indices` when destination pages are fewer.
   - Inference: a prefill/decode page-position mismatch can drop KV pages and still continue, which fits bad decode output symptoms better than a local shape mismatch that would fail-fast.

2. EAGLE CP-local side-channel gate is not tied to actual metadata creation.
   - Evidence: `_can_use_cp_draft_shared_kv()` checks env/model capability, not `forward_batch.nsa_cp_metadata`; target metadata is created only under `can_cp_split(...)`.
   - Inference: unusual extend/padding/can-split edge cases can still mark hidden as CP-local without a fully proven CP metadata contract.
   - Lower than risk 1 because the stale post-capture marker was fixed and bs>1 hidden mismatches fail-fast.

3. HiCache chunked post-split stale-tail prune can still escape the deferred-insert path.
   - Evidence: `insert()` calls `_cp_prune_stale_tail_after_page_floor()` after `_split_node()` even after pre-checks.
   - Inference: chunked prefill plus in-flight backup can still hit uncaught `HiCachePendingBackupSplit` in rare timing windows.

4. bs>1 partial-current prefix span can fall off TAI IPC and over-communicate.
   - Evidence: batch prefix span may include contiguous slots beyond pure prefix; TAI IPC materialize fast path requires `start_slot == 0`.
   - Inference: this is primarily a performance/regression risk and a complexity hotspot, not a proven correctness bug because current rows are later overwritten/masked.

5. bs>1 async prefetch fast path is not implemented.
   - Evidence: both MLA and index prefetcher creation skip `batch_size != 1`.
   - Inference: short-extend batching can improve compute occupancy, but cache-hit paths may still pay synchronous L2->L1 materialize/collective costs.

6. Runtime compatibility fingerprint is too narrow.
   - Evidence: fingerprint excludes several files that define CP shared-KV/runtime/EAGLE behavior and excludes tai-kernel version.
   - Inference: prefill/decode or host/container mixed-version runs can pass bootstrap while using incompatible CP contracts.

7. Allocation/transfer contiguity is still best-effort.
   - Evidence: owner-lane allocator takes sorted bucket prefixes; Mooncake groups contiguous runs and bandwidth depends on group count.
   - Inference: this explains low transfer bandwidth under fragmented long-running workloads, but it is not the strongest garbage-output root cause.

Checked-and-downranked items:

- Dummy page 0 semantics are explicit and preserved by `CpSharedKVLayout`; current evidence does not point to page0 as the main issue.
- Direct-write local `out_cache_loc` now uses valid split rows and fail-fast checks; current evidence does not point to silent direct-write row misalignment.
- Current-index fp8 dtype/local-base issue was repaired in `f50e2b1e0` and has unit coverage; it remains a runtime item to verify, not the leading code-level suspect in this pass.

Next discriminating probes/fixes to prioritize:

1. Replace Mooncake CP shared-KV transfer truncation with fail-fast under CP shared-KV and add a focused unit test for mismatched logical positions vs destination pages.
2. Tie EAGLE CP-local draft enablement to actual `forward_batch.nsa_cp_metadata` / batch plan availability, or fail-fast when the marker is set without metadata.
3. Add a regression for HiCache chunked insertion where a stale tail is blocked by pending backup; expected behavior is deferred insert/rollback, not scheduler crash.
4. Extend runtime fingerprint to include CP shared-KV runtime/prefetch/EAGLE/NextN and tai-kernel version/build identity.
5. For performance work, measure bs>1 partial-current sync compose and transfer contiguous-group counts before changing allocator policy again.

## 2026-06-04 Mooncake transfer page-count fail-fast implementation

Change made:

- Added `validate_transfer_page_count_or_raise()` in `python/sglang/srt/disaggregation/utils.py`.
- Replaced the Mooncake KV transfer truncation workaround in `python/sglang/srt/disaggregation/mooncake/conn.py` with the new validator.
- Added unit coverage in `test/registered/unit/disaggregation/test_cp_shared_kv_transfer_mapping.py`.

Contract:

- Source prefill pages and selected decode destination pages must have exactly the same count.
- Under CP shared-KV, a mismatch now raises `[CP_SHARED_KV_FAIL_FAST][mooncake_transfer_page_count_mismatch]` with room, cp_rank, source/destination lengths, logical positions, index slice, and compact page summaries.
- The old behavior truncated `prefill_kv_indices` to the destination length. That is no longer allowed because it can hide prefill/decode page-map corruption and let decode continue with incomplete KV.

Verification status:

- Local `py_compile` passed for `disaggregation/utils.py`, `disaggregation/mooncake/conn.py`, and the updated unit test.
- Local pytest is blocked by missing runtime dependency `orjson` in this environment.
- Remote RED/GREEN execution could not be completed in this pass because `ssh g0034` timed out.

## 2026-06-05 当前现象修正：decode 已结束但输出仍乱码/重复，accept len=4 不是恢复正确性

远端最新日志：

- prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260604_181200.log`
- decode0: `/mnt/beegfs/cjy/log/decode0_20260604_181227.log`
- decode1: `/mnt/beegfs/cjy/log/decode1_20260604_181237.log`

新增事实：

1. 用户确认：请求能返回结束，但内容依然重复、无意义。这说明当前主症状不是“scheduler 不停止”，而是模型 token 分布/上下文状态已经损坏。
2. decode1 在 18:17:52-18:19:29 持续显示 `accept len: 4.00, accept rate: 1.00`，并且 `#transfer-req: 0`、`#retracted-req: 0`。这不是之前的 accept collapse；它表示 draft 4 个 token 基本全部被 target 接受。若输出乱码，说明 target 验证路径本身也在错误上下文上同意了错误 token。
3. decode warmup 本身已经异常：`prompt_tokens=4` 的 warmup 输出固定为 `! of the of the of the`，`completion_tokens=8`，`spec_accept_rate=0.25`。这发生在 40k cache-hit 之前，因此不能继续把问题限定为长 prompt/HiCache cache-hit。
4. prefill warmup 同一阶段输出是 `text=' /'`，而 decode warmup 输出重复 `of the`。这把范围收窄到“prefill 产生并交给 decode 的 persistent KV / disagg transfer / decode FP8 KV/EAGLE 消费路径”，而不是 prefill 直接 forward 的 logits 路径。
5. 日志中未看到 `CP_SHARED_KV_FAIL_FAST` transfer page-count mismatch。说明最新 page-count fail-fast 没抓到这个问题；如果仍是 transfer 问题，更可能是内容/order/scale 级别，而不是页数截断。
6. decode 启动时所有 DP rank 打印：`Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0.` 该风险仍未排除。FP8 KV 下若 scale/packed layout/transfer bytes 不一致，target 与 draft 都可能在同一个错误 KV 上稳定地产生重复 token。

当前重新排序的假设：

- H1（最高）：CP prefill persistent KV 在 FP8 下写入/transfer/decode 接收的内容、顺序或 scale 合同不一致。page-count fail-fast 只能排除页数截断，不能排除每页内容错位、FP8 scale 行错位、或 decode 端按不同 layout 解释。
- H2：decode FP8 KV cache / EAGLE 在没有 scale factors 时本身就会产生退化输出；需要用同一 disagg 配置但禁用 FP8 KV 或禁用 EAGLE 做最小分叉验证。
- H3：长 prompt/HiCache cache-hit 不是当前首要根因，因为 4-token decode warmup 已异常。

禁止重复误区：

- 不要再把 `accept len=4.00` 当成“恢复正常”；它只能说明 speculative accept 高，不代表 token 语义正确。
- 不要只盯 40k cache-hit；短 warmup 已经提供更小复现面。
- 不要继续只检查 page-count mismatch；当前 fail-fast 未触发，下一步应查 FP8 packed KV 的 byte layout、scale、source/destination page order。

## 2026-06-05 cache-hit 口径核对：当前日志里的首个 40k prefill 请求未命中

用户追问“这次第一次请求为什么会 cache hit”。对当前已检查的 prefill 日志 `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260604_181200.log`：

- server warmup 请求在 18:15:17，`prompt_tokens=4`，prefill batch 显示 `#cached-token: 0`，但会插入/backup 一个 `logical_len=3` 的小 radix/HiCache node。
- 40k 长请求在 18:16:41，prefill batch 行 1020-1027 显示 `#new-token: 40448, #cached-token: 0`。
- 因此这份日志中，首个长业务请求没有 prefill cache hit。若客户端或另一轮日志显示“第一次请求 cache hit”，需要区分：
  1. 该 prefill 进程是否真的重启/清 cache；radix/HiCache 是进程内状态，client/router 重启不会清。
  2. replay 的“seq 第一条”不等于 scheduler 实际第一条；并发/重试可能先插入相同 prefix。
  3. `cached_tokens` 客户端字段与 prefill log 的 `#cached-token` 可能不是同一条请求/同一统计口径。

下一步如果再次出现“首条命中”，应先抓对应 log 文件和请求时间，核对该请求之前是否已有同 prompt/prefix 的 `POST /v1/chat` 或 warmup/replay 请求插入。

## 2026-06-05 当前最高置信根因：bs=1 batch-plan 在非 page-aligned tail 下误判未启用 compute padding

新增代码证据：

- `python/sglang/srt/layers/attention/nsa/utils.py:2439-2457` 在 CP shared-KV 下即使 `batch_size=1` 也会构造 `CPSharedKVBatchPlan`。
- `python/sglang/srt/layers/attention/nsa/utils.py:604-676` 同时构造两套 split：
  - `request_split_lists` / valid split：只覆盖真实 token，tail page 的最后一个 segment 可能小于整页。
  - `request_compute_split_lists` / compute split：以 page 为最小单位，tail page 按整页参与 attention/index compute。
- `python/sglang/srt/layers/attention/nsa/utils.py:718-723` 旧判定用 `request_compute_padded_tokens != request_valid_padded_tokens` 决定 `compute_padding_enabled`。这只能发现“为了覆盖至少 cp_size 个 page 而额外 pad”的场景，无法发现“总 page 数相同但最后一个 segment 从 tail token 扩到整页”的场景。
- 远端最小复现：
  - `extend_len=40387, page_size=64, cp_size=8, cp_rank=0`
  - valid local rows = `2560 + 2435 = 4995`
  - compute local rows = `2560 + 2496 = 5056`
  - 旧代码却给出 `compute_padding_enabled=False`
  - `40387` 正是当前业务请求的 `prompt_tokens`，因此这是当前乱码输出的强相关输入类。

影响链路：

1. `split_tensor_by_cp_batch_plan(..., split_kind="compute")` 在 `compute_padding_enabled=False` 时会选 valid split，返回 4995 行。
2. batch plan 内的 `request_actual_seq_q_prev/next`、`request_compute_seq_q_prev/next` 已按 compute split 记录 5056 行的语义。
3. 于是同一个 forward 内存在“tensor 实际 CP-local 行数是 valid rows，但 metadata/attention/index 部分消费者认为是 compute rows”的不一致。
4. index top-k 路径已有 `_select_batch_topk_query_lengths()` 的局部自适应逻辑，能在某些地方避免 fail-fast；但 attention/MLA/cache write 仍可能在不一致 metadata 下写入或读取错误 CP-local row，导致 prefill 生成错误 KV/logits，decode 在错误上下文上高 accept 但输出乱码。

结论：

- 这比 FP8 scale、Mooncake page-count mismatch 更贴近当前现象：它只在 non-page-aligned 长 prompt 下触发，能解释为什么 `prompt_tokens=40387` 的请求输出不说人话，同时 fail-fast page-count 没有触发。
- 修复方向不是扩大 fallback，而是让 `compute_padding_enabled` 表示“valid split 与 compute split 是否不同”，而不是只比较 padded-token 总量。
- 回归测试必须覆盖 `40387/64/CP8`：期望 `compute_padding_enabled=True`，compute split 返回 5056 local rows，cache write valid selector 过滤回 4995 rows。

## 2026-06-05 修复记录：compute_padding_enabled 改为 split-shape 语义

代码修复：

- `build_batch_page_aligned_in_seq_split_plan()` 的 `compute_padding_enabled` 从“比较 valid/compute padded-token 总量”改为“比较 valid/compute split list 是否完全一致”。
- 这样 `extend_len=40387,page_size=64,cp_size=8` 会正确进入 compute-padding 路径：
  - CP rank0 compute local rows = 5056。
  - cache write valid selector 继续过滤回 valid local rows = 4995。
- 保留 page-aligned、valid split 与 compute split 完全相同的场景为 `compute_padding_enabled=False`；对应 static padding 测试已改为 `extend_len=8,page_size=4,cp_size=2`，避免继续把 non-page-aligned tail 误认为“无 compute padding”。

验证：

- 本地 `py_compile` 通过：`utils.py`、`test_nsa_cp_utils.py`。
- 本地 pytest 仍被环境缺失 `orjson` 阻塞。
- 远端 `g0034` 容器内通过：
  - `python -m py_compile python/sglang/srt/layers/attention/nsa/utils.py test/registered/unit/layers/test_nsa_cp_utils.py`
  - 4 个针对性回归测试：`4 passed`
  - 全量 `test/registered/unit/layers/test_nsa_cp_utils.py`：`87 passed`

剩余验证缺口：

- 还需要用户用新 prefill 进程重新打同一个 40387-token 请求，确认输出是否恢复；当前修复解释了现象，但 ETE 仍必须验证。
- FP8 scale / Mooncake 内容顺序问题仍是次级假设；如果该修复后仍乱码，下一轮应继续查 FP8 packed KV byte layout 与 transfer page order，而不是回到 page-count 或 cache-hit 口径。

## 2026-06-05 再次乱码后的新定位：current-only index current reuse TAI fast path 会静默补零

最新事实：

- compute-padding split-shape 修复后，用户重新验证仍然输出乱码/重复，说明上一轮修复只解决了一个真实不一致，但不是唯一根因。
- 最新 prefill 日志中的 40k 首请求仍是 cache miss（`#cached-token: 0`），所以当前错误必须能发生在 current-only extend 路径。

新增代码证据：

1. `nsa_indexer.py::_can_reuse_current_index_kv()` 在 CP shared-KV + current reuse 开启时，对 current-only extend 也返回 true。
2. `nsa_indexer.py::forward_cuda()` 随后用 `select_cp_local_valid_rows_for_cache_write()` 准备 `current_index_kv`。在 CP shared-KV batch plan 下，这个 tensor 是当前 CP rank 的 owner-lane compact rows，不是 request 的全局 dense rows。
3. `nsa_indexer.py::_get_topk_in_seq_cp_pair_batch()` 对 `current_only_batch` 不调用 `_maybe_materialize_shared_index_buffer()`，而是把 compact `current_index_kv` 直接传入 `_get_topk_ragged_with_cp()`。
4. `_get_topk_ragged_with_cp()` 的 TAI current-index batch descriptor 使用 `kv_lens_list=end_seq_position`。对首个 40k current-only 请求，segment 的 `kv_len` 是全局序列长度量级；但 `current_bases_list` 来自本 rank local offset，`current_index_kv` 行数只有本 rank owner rows（例如 rank0 约 4995/5056 行）。
5. `tai-kernel/python/tai_kernel/nsa_prefill/cp_index_mqa_prepare.py::_prepare_cp_mqa_current_kv_batch_kernel` 的 mask 是 `src_tokens < CURRENT_ROWS`。当 `kv_len` 大于 compact local rows 时，超出的全局 K 行被静默写成 0，而不是 fail-fast。

结论：

- 这解释了“uint8 修复后不再 fallback/报错，但模型仍不说人话”：以前 TAI current-index path 因 dtype 不匹配失败并回退；修复 dtype 后快路径开始成功执行，但它把 CP-local compact current K 当成 global dense K 使用，导致 top-k 的 index K 大量补零。
- 该问题能直接污染 target index/top-k，进而让 target attention 在错误上下文下生成/接受错误 token；这比 decode 端 EAGLE 单独异常更符合 `accept len` 高但输出乱码的现象。

当前修复策略：

- 暂时禁用 **CP shared-KV current-only 的 compact current-index reuse**，并打印醒目的 fallback warning。
- current-only index top-k 改回 full shared index materialize 路径：先写入各 CP owner page，再通过 shared paged buffer materialize/all-reduce 得到全局 dense index K。
- partial-current（有 page-aligned prefix）仍保留现有 prefix+current compose 路径；它有 prefix page contract 和 current loc shape 校验，另行验证。

后续如果要重新打开 current-only fast path，必须新增 descriptor，显式表达“每个 CP rank 的 local owner rows 如何映射到全局 token positions”，不能继续用 `current_base + token_offsets` 当 global dense row。

## 2026-06-05 修复记录：禁用 current-only compact index current reuse

代码修复：

- `Indexer._can_reuse_current_index_kv()` 在 current-only extend 下返回 false，并打印一次 `[CP_SHARED_KV_FALLBACK][index_current_reuse] reason=current_only_compact_unsupported`。
- `_get_topk_in_seq_cp_pair()` 与 `_get_topk_in_seq_cp_pair_batch()` 增加防御：即使调用者误传 compact `current_index_kv`，current-only CP shared-KV index top-k 也会忽略它，改走 full shared index materialize，并打印一次 `current_only_compact_ignored`。
- partial-current index reuse 不变：有正的 page-aligned prefix 时仍使用 `_maybe_materialize_shared_index_buffer(..., current_index_kv=...)` 的 prefix+current compose。

为什么这次修复针对当前乱码：

- current-only cache-miss 40k 请求会先写各 CP owner page 的 index K，再做 full shared materialize；不会再把本 rank compact rows 当作全局 dense rows 喂给 TAI current-index top-k kernel。
- 这避免了 TAI kernel 用 `src_tokens < CURRENT_ROWS` 静默把大部分 global K rows 填 0 的问题。

验证：

- 本地 `py_compile` 通过：`nsa_indexer.py`、`test_nsa_cp_utils.py`、`test_cp_shared_kv_runtime.py`。
- 本地 pytest 仍被缺失 `orjson` 阻塞。
- 远端 `g0034` 容器内通过：
  - `python -m py_compile python/sglang/srt/layers/attention/nsa/nsa_indexer.py test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py`
  - 针对性 3 个回归：`3 passed`
  - `test_nsa_cp_utils.py` + `test_cp_shared_kv_runtime.py` 全量：`196 passed, 2 subtests passed`

剩余验证缺口：

- 还需要用重新启动后的 prefill/route 流量验证 40387-token 请求输出是否恢复。
- 如果仍乱码，下一轮应继续查 FP8 MLA/current reuse 或 transfer 内容顺序；但当前 index current-only 静默补零已经是明确 bug，不能再回到该路径。

## 2026-06-05 bs>1 改造后 bs=1 也坏的回归链路

最新用户反馈：index current-only gate 生效后，输出仍然不说人话，并且 decode accept len 很低。最新日志确认：

- prefill 首个 40k 请求仍是 cache miss：`#new-token: 40448, #cached-token: 0`。
- decode 端该请求 accept len 约 `1.0~1.5`，不是高 accept 误接收，而是 draft/target 在错误上下文下明显不一致。
- `index_current_reuse` 的 current-only compact gate 已在 warmup 生效，说明远端代码不是旧版本。

新的关键差异：

1. bs>1 改造后，`utils.py` 的 CP metadata 入口只要 `uses_cp_shared_kv` 且存在 `extend_seq_lens_cpu/extend_prefix_lens_cpu`，即使 `batch_size=1` 也会构造 `CPSharedKVBatchPlan`，不再走旧 scalar bs=1 path。
2. commit `50d000870` 去掉了 current reuse 的 `batch_size == 1` 保守限制，并把 current-only extend 直接判定为可 reuse。
3. 旧 scalar bs=1 下，`current_extend_kv_rows_for_reuse()` 用 global `extend_len` 校验 tensor rows；CP split 后的 `k/k_rope` 通常只有本 rank local rows，因此 current reuse 不会启用，会走 full shared materialize。
4. 新 batch/compute-padding path 下，`nsa_backend.py` 先用 `select_cp_local_valid_rows_for_cache_write()` 得到本 rank owner-local current rows，再用 `local_out_cache_loc.numel()` 校验，因此 current-only MLA reuse 会被启用。

当前最高置信破坏点：MLA/attention current-only page-slot compose。

- `materialize_prefix_and_reuse_current_kv_page_slots(prefix_pages=0)` 会创建 dense slot buffer。
- prefix 为 0 时，不会 materialize/reduce 任何 prefix rows。
- 随后 `fill_current_kv_page_slots_and_remap_locs()` 只把本 rank owner-local current rows 填进 dense slot。
- 目前该函数不会对 current/suffix page slots 做 all-reduce/all-gather。
- 因此 attention 可能看到：本 rank owner pages 有值，其他 rank owner pages 是 0/stale，直接污染 prefill KV/logits。

这解释了为什么“之前 bs=1 可以，现在 bs>1 后同样 bs=1 错”：不是 scheduler 真组了 bs>1，而是 bs>1 metadata/current-reuse 改造改变了 bs=1 的 fast path，使其开始走 owner-local current reuse。

下一步验证：

- 增加单测证明 `materialize_prefix_and_reuse_current_kv_page_slots(prefix_pages=0)` 对 cp_size>1 且 current_locs 只含本 rank owner rows 时，不会聚合其他 owner pages。
- 修复方向二选一：
  1. 保守 correctness：CP shared-KV current-only MLA reuse 先禁用，回到 full shared materialize。
  2. 正式 fast path：填 current rows 后，对 current page slots 执行 owner-aware 聚合，仅 reduce/gather current slots，不能重复 reduce 已经 reduced 的 prefix slots。

短期为了恢复正确性，优先做 1；随后再单独实现 2 以恢复性能。

## 2026-06-05 修复记录：current-only MLA/current reuse 回退到 full shared materialize

短期 correctness 修复：

- `should_reuse_current_extend_kv()` 在 `uses_cp_shared_kv=True` 且 current-only extend 时返回 false。
- 新增 warning：`[CP_SHARED_KV_FALLBACK][current_reuse] reason=current_only_owner_local_unsupported`。
- 这样 current-only cache-miss 请求不会再进入 MLA current page-slot compose，而是走 full shared materialize：先从 owner-local persistent pool materialize owned pages，再通过 collective/IPC 得到每个 rank 完整 dense KV view。

为什么这能解释并修复 bs>1 后 bs=1 回归：

- bs>1 改造让 bs=1 也构造 `CPSharedKVBatchPlan`，并在 compute-padding path 中把 current KV rows 收窄为本 rank owner-local compact rows。
- current-only MLA reuse 没有 current page-slot 聚合，attention 会消费不完整 dense KV。
- 禁用 CP shared current-only reuse 后，bs=1 cache miss 回到旧的 full materialize 消费语义，避免 owner-local compact rows 被当作 global dense rows。

保留内容：

- `is_current_only_extend_batch()` 本身不变，它仍是描述性 helper。
- 非 CP shared-KV 的 current-only reuse 不受影响。
- partial-current reuse 暂未关闭；如果后续 cache-hit 仍有异常，需要继续实现 current suffix page-slot 聚合或临时关闭 partial-current。

验证：

- 本地 `py_compile` 通过：`cp_shared_kv_runtime.py`、`test_cp_shared_kv_runtime.py`。
- 远端 `g0034` 容器内通过：
  - 新增 targeted test：`1 passed`
  - `test_cp_shared_kv_runtime.py`：`110 passed, 5 warnings, 2 subtests passed`
  - `test_nsa_cp_utils.py test_cp_shared_kv_runtime.py`：`197 passed, 5 warnings, 2 subtests passed`

剩余验证缺口：

- 需要用户重启 prefill 后重新打 40387-token 请求，确认输出和 decode accept len 是否恢复。
- 若 current-only 恢复但 cache-hit partial-current 仍坏，下一步应实现“fill current rows 后只聚合 current page slots”的正式 fast path，而不是继续让 owner-local compact current rows直接给 attention 消费。

## 8. 2026-06-05 继续排查：current-only MLA 禁用已生效但乱码仍存在

新增证据来自：

- prefill：`/mnt/beegfs/cjy/log/sglang_cp_hicache_20260604_200924.log`
- decode：`/mnt/beegfs/cjy/log/decode0_20260604_200912.log`

确认事实：

1. 最新 prefill 进程确实使用了当前代码：日志 885-952 多次出现
   `[CP_SHARED_KV_FALLBACK][current_reuse] reason=current_only_owner_local_unsupported`。
   因此“current-only MLA/current reuse 仍在启用”不是最新复现的主因。
2. decode 侧 warmup 在 prefill 20:12:48 ready 之前（20:11:20）已经输出异常：
   `!!!!!!!??!??!` / `! of the of the of the`，并且 spec accept rate 只有 0.13-0.25。
   这说明当前坏现象至少不只来自长 prompt HiCache 命中或 Mooncake transfer；decode 自身/共享代码路径/FP8 KV cache 也需要被纳入排查。
3. prefill warmup 对 4-token request 的 CP shared-KV path 中，`current_only_owner_local_unsupported` 生效后仍反复触发
   `[CP_SHARED_KV_FALLBACK][tai_index_mqa_prepare] reason=batch_kernel_failed ... error=descriptor tensors must share dtype segments=1 total_kv_len=4 total_q_count=4`。
   这说明 index MQA prepare fast path 在 FP8/当前 dtype 组合下没有工作，但尚未证明 fallback 本身会破坏语义。
4. 长 40k 请求仍有 `tai_index_mqa_prepare` fallback：`segments=2 total_kv_len=43072..43456 total_q_count=5056`。
   当前不能再把问题只归咎到 current-only reuse；下一步需要检查 full materialize、persistent KV direct write、FP8 KV store/load、decode flashmla_kv 的共同合同。
5. `omx explore` 在本机因 Cargo.lock version 4 解析失败，后续本轮改用普通 grep/sed；该失败不是模型路径证据。

当前高优先级假设（待验证）：

- H1：FP8 KV cache 的 persistent store 或 full materialize page-slot compose 写入/读取的 dtype/layout 与 flashmla_kv decode 消费合同不一致。支持证据是 decode warmup 已异常，且当前运行使用 `kv_cache_dtype='fp8_e4m3'`、decode backend `flashmla_kv`。
- H2：`tai_index_mqa_prepare` fast path 因 descriptor dtype 不一致失败后，Python per-segment fallback 在 compute padding / page-tail / FP8 下构造的 K/S 或 top-k order 不等价。支持证据是所有最新复现都有该 warning；反证是该 warning 可能只是性能 fallback，需要单测或最小 CPU/GPU 对比证明。
- H3：CP shared-KV full materialize 对 current-only prefix=0 读 persistent pool，而 persistent pool direct write 在当前流/TAI fused store/FP8 packed layout 下未完成或写错。支持证据是 `should_reuse_current_extend_kv=False` 后会从当前 k/k_rope fast path退到读 pool。

已排除/降级：

- 最新复现不再优先怀疑 current-only MLA/current reuse 本身，因为对应 warning 已证明它被禁用。
- 最新复现不应只盯长 prompt/HiCache/transfer；decode warmup 异常把问题范围扩大到 decode 或共享 FP8 attention 路径。

## 2026-06-05 继续排查：最新 ETE 仍乱码后的新增事实

最新远端日志：

- prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260604_200924.log`
- decode: `/mnt/beegfs/cjy/log/decode0_20260604_200912.log`

新增证据：

1. 最新 prefill 仍在 40k 首次请求上走 cache miss / current-only 场景：
   - `Requested token count (40391 input + 200000 new)`；
   - `Prefill batch, #new-seq: 1, #new-token: 40448, #cached-token: 0`；
   - decode 后续 `#token: 40448/40512, accept len: 1.43/1.38`。

2. `current-only MLA/current reuse` 禁用逻辑确实已生效：日志出现
   `[CP_SHARED_KV_FALLBACK][current_reuse] reason=current_only_owner_local_unsupported ... Falling back to full shared materialize`。
   因此“旧 current-only reuse 还在直接 corrupt”的假设降低优先级。

3. 远端仍有大量 `tai_index_mqa_prepare` fallback：
   `error=descriptor tensors must share dtype segments=2 total_kv_len=43072..43456 total_q_count=5056`。
   代码检查确认：`nsa_indexer.py::_get_topk_ragged_with_cp()` 的 non-current batch fast path把 `batch_indices` 构造成 `int64`，而 `kv_lens/q_starts/q_lens/k_bases/q_bases` 是 `int32`；TAI C++ kernel 要求这些 descriptor tensors 同 dtype。
   这解释了该 warning。它是明确性能/覆盖问题，但 fallback 路径仍应语义正确，暂不单独作为“乱码”的根因结论。

4. TAI IPC slot-dense materialize 的 dummy page offset 已检查：
   - `tai-kernel/python/tai_kernel/nsa_prefill/ipc.py` 文档声明 slot `i` 写 dense page `i+1`；
   - `kvcacheio_lf_pf.cu::ipc_materialize_peer_pages_slot_dense_*` 实现使用 `dense_page = slot + 1`，并 zero page 0。
   因此“IPC kernel 把 page 0/dummy page 写错导致整体 shift”的假设已排除。

5. 新的更强 correctness 风险：`SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1` 下，`materialize_shared_token_kv_buffer()` 如果 `_try_tai_ipc_materialize_token_kv_page_slots_into()` 成功，会跳过 `_all_reduce_materialized_buffer()`。
   对 prefix/stable cache page 这可能成立；但对当前 cache miss 的 current-only 请求，MLA direct write 刚把本层 owner-local KV 写入各 rank 的 pool，随后同一层 attention 立即通过 IPC 读取 peer rank 的 pool。这里没有 per-layer cross-rank event/barrier 保证 peer rank 的 direct-write kernel 已完成。
   旧 all-reduce materialize 至少通过 collective 提供跨 rank stream/order 同步；IPC peer read 跳过 collective 后可能读到未写完或旧 KV。这与“prefill HTTP 200、decode 消费 persistent KV 后 accept len≈1/输出乱码”高度匹配。

当前优先假设：

- H1': 当前-layer fresh KV 的 CP full materialize 不能使用无同步的 TAI IPC peer read。需要先在 current-only/fresh-current 场景禁用 IPC materialize，回到 local materialize + all-reduce；后续如果要保留 IPC，需要设计显式跨 rank write-complete 同步或恢复正确 current-reuse dense compose。

待验证/修复：

- 给 `materialize_shared_token_kv_buffer()` 增加 `allow_ipc_materialize` 控制；在 MLA full materialize callsite 对 `is_current_only_extend_batch(forward_batch)` 传 `False`。
- 同时修正 `tai_index_mqa_prepare` non-current batch descriptor dtype：`batch_indices` 应与其他 descriptor 同 dtype，优先 `int32`。
- 补单测：current-only MLA full materialize callsite 必须禁止 IPC；non-current TAI batch descriptor dtype 必须统一。

## 2026-06-05 修正：不做 blanket disable，必须修 fresh-current 合同

用户明确指出：不能盲目禁止 fast path；必须找到问题并尝试修复。

纠正当前方向：

- 不把 `SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE` 全局关掉，也不简单在 current-only 下禁止 IPC 作为最终方案。
- 真正需要验证的是 fresh current KV 的 source-of-truth：同一层 forward 里刚由 direct-write 写入 persistent pool 的 current KV，是否可以立即作为 full-materialize 的稳定读源。
- 若问题成立，正确修复应是 owner-aware current compose：prefix/stable pages 仍可用 IPC/collective materialize，当前 extend rows 从本层已有的 CP-local `k/k_rope` 源张量按 logical loc 填入 dense slot，并在必要时只同步缺失 owner pages；不能简单退回慢路径。

下一步检查：

1. 阅读 `materialize_prefix_and_reuse_current_kv_page_slots()` 与 `fill_current_token_kv_page_slots_and_remap_locs()`，确认它是否已经具备 owner-local current row 填充合同。
2. 确认当前为什么禁用了 current-only reuse：是缺少全 rank current rows，还是调用侧传错 loc/base。
3. 构造一个最小单测验证：current-only CP shared KV 不应从 persistent `kv_cache` 读取 current pages；应从 current source tensor 填入 dense page slots。这个测试先 RED，再修。

## 2026-06-05 修正方向：不禁用 current fast path，修 owner-local current compose 后的同步合同

用户指出不能继续靠盲目禁用定位。重新审视后，当前修复方向调整为恢复 current-only fast path，并修正它的真实缺口：

- current-only / partial-current compose 的 source-of-truth 应该是当前 forward 产生的 owner-local `k/k_rope` 或 index K，而不是刚写入 persistent pool 的 fresh current pages。
- 但是 owner-local current rows 填入 slot-dense buffer 后，原实现没有对 current slots 做 CP 同步；每个 rank 只能看到自己 owner lane 的 current pages，其它 owner pages 仍是 0/stale。
- 这会同时影响 MLA attention current reuse、index current reuse，以及 prefetch consume-prefix-with-current 路径。

本轮代码修复：

1. 新增 `build_batch_current_slot_spans()`：用 CPU `prefix_lens_cpu/extend_lens_cpu` 计算 current page slot spans，避免扫描 CUDA page table。
2. `materialize_prefix_and_reuse_current_kv_page_slots()`：填入 current rows 后，只对 current token slot rows 做 `_all_reduce_materialized_buffer_range()`；prefix rows 不重复 reduce。
3. `materialize_prefix_and_reuse_current_index_page_slots()`：同样只对 current index page slots 做 range reduce。
4. `CpSharedKVMlaPrefetcher.consume_prefix_with_current()` 与 `CpSharedKVIndexPrefetcher.consume_prefix_with_current()`：prefetch 已经完成 prefix reduce/IPC 后，consume 时填 current rows，再对 suffix/current slots 做同步。
5. `Indexer._can_reuse_current_index_kv()` 不再 blanket 禁用 current-only；`_maybe_materialize_shared_index_buffer()` 允许 zero-prefix current-only compose，并把 `current_index_kv` 传入 compose。
6. 修复 TAI non-current index batch descriptor dtype：`batch_indices` 从 `int64` 改为 `int32`，与 `kv_lens/q_starts/q_lens/k_bases/q_bases` 保持一致，避免 `descriptor tensors must share dtype` fast-path fallback。

为什么这不是盲目禁用：

- 没有关 `SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE`。
- 没有关 current-only MLA/index reuse。
- 修复的是 owner-local current rows 填入 dense slot 后缺少跨 CP rank 聚合这一明确合同漏洞。

验证：

- 本地 `py_compile` 通过：`cp_shared_kv_runtime.py`、`nsa_indexer.py`、`nsa_backend.py`、`cp_shared_kv_prefetch.py`、相关单测文件。
- 远端 `g0034` 容器内 `py_compile` 通过同一组文件。
- 远端 targeted runtime tests 通过：current slot span、token current compose reduce、index current compose reduce、owner-local current rows gate。
- 远端完整 `test_cp_shared_kv_runtime.py` 通过：`111 passed, 5 warnings, 2 subtests passed`。
- `test_nsa_cp_utils.py` 当前远端 pytest collection 被环境里的 `sgl_kernel`/fake-op 注册问题阻塞；已用直接 Python 脚本验证：
  - TAI shared batch descriptor `batch_indices.dtype == torch.int32`。
  - current-only index path 会把 `current_index_kv` 交给 materialize/compose，而不是丢弃后读 persistent full materialize。

剩余验证缺口：

- 需要用户重启 prefill 并重新打业务请求，确认 40387-token 请求输出与 decode accept len 是否恢复。
- 如果仍乱码，下一步继续查 FP8 packed MLA KV compose/transfer byte layout，而不是回退到禁用 current fast path。

### 2026-06-05 追加验证

本轮继续验证不采用 blanket disable 的修复：

- 远端 `g0034` 容器内 `py_compile` 再次通过：
  `cp_shared_kv_runtime.py`、`nsa_indexer.py`、`nsa_backend.py`、`cp_shared_kv_prefetch.py`、`test_cp_shared_kv_runtime.py`、`test_nsa_cp_utils.py`。
- 远端 `test_cp_shared_kv_runtime.py` 完整通过：
  `111 passed, 5 warnings, 2 subtests passed`。
- 远端 targeted tests 通过：
  - `test_batch_current_slot_spans_follow_prefix_and_extend_pages`
  - `test_materialize_prefix_current_token_kv_uses_ipc_and_reduces_current_slots`
  - `test_materialize_prefix_current_index_uses_ipc_and_reduces_current_slots`
  - `test_cp_shared_current_only_reuse_accepts_owner_local_current_rows`
- 新增 direct 脚本验证 `Indexer._maybe_materialize_shared_index_buffer()` 对 current-only index compose 的合同：
  - `prefix_pages == 0`
  - `current_slot_spans == [(0, 2)]`
  - `current_index_k/current_index_scale` 直接来自当前 forward source tensor
  - 输出 `CURRENT_ONLY_INDEX_MATERIALIZE_OK`

注意：

- `test_nsa_cp_utils.py` 与 runtime tests 混跑时，远端 pytest collection 仍可能被当前环境的 `sgl_kernel` 安装不完整阻塞：
  `cannot import name 'gelu_tanh_and_mul' from 'sgl_kernel'`。
  这是测试环境 import 问题，不作为本轮语义验证失败；关键合同用 targeted/direct 验证覆盖。

## 2026-06-05 regression focus update: baseline is pre-bs>1, but split-shape compute-padding fix is already in HEAD

User-confirmed baseline: before the bs>1 CP shared-KV work, both BF16 KV cache and FP8 E4M3 KV cache produced sane output. Therefore the current garbage-output issue remains a bs>1-series regression until proven otherwise.

Correction to avoid repeated re-checking:

- The earlier concrete `40387/page-tail compute_padding_enabled` bug has already been fixed in HEAD: `compute_padding_enabled` is now based on valid-vs-compute split shape, so non-page-aligned long prompts enter the explicit compute-padding contract instead of mixing valid tensors with compute metadata.
- The just-committed current-slot fix also preserves MLA/index current reuse and TAI materialize; it does not disable fast paths.
- The latest remote logs available at `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260604_221557.log` show the 40k bad request at `22:23:22`, which is before local commit `c8510938b` (`22:25:06 UTC-equivalent log time`). After that commit, the log only shows health/warmup 64-token traffic, not a new 40k business repro.

Next investigation should not re-open the already-fixed split-shape bug unless a post-`c8510938b` business repro shows the same metadata inconsistency. The next code-level audit should focus on bs>1-series changes that still affect bs=1 despite the split-shape/current-slot fixes: EAGLE/draft side-channel gating, CP batch-plan vs scalar-path differences, and FP8 packed KV materialize/transfer byte-order/scale contracts.

## 2026-06-05 user-confirmed regression boundary

User explicitly confirmed: before the bs>1 CP shared-KV work, both FP8 E4M3 KV cache and BF16 KV cache were correct. Treat this as the regression boundary.

Implication:

- Do not spend more cycles treating FP8/BF16 support itself as the primary cause unless new evidence contradicts this boundary.
- The active suspect set is bs>1-era code that also affects bs=1: batch-plan metadata replacing scalar CP metadata, compute-padding/page-tail handling, current/partial reuse changes, and EAGLE/draft side-channel interaction with the new CP metadata.
- A valid fix should preserve the pre-bs>1 FP8/BF16 fast paths and repair the bs>1-induced semantic difference, not blanket-disable MLA/index cache reuse.

## 2026-06-05 correction: bs>1-era compute padding over-applies to normal long page tails

User confirmed the regression boundary: before the bs>1 work, both FP8 and BF16 paths were correct.  That makes the current primary suspect the bs>1 batch-plan / compute-padding path changing the bs=1 prefill semantics, not FP8/BF16 storage itself.

New RED tests on remote `g0034` show current production code treats `extend_len=40387,page_size=64,cp_size=8` as compute-padded:

- valid local rows on CP rank0: `4995`
- compute local rows on CP rank0: `5056`
- `compute_padding_enabled=True`

This is now considered an over-correction.  A normal long request with only a final partial page should keep compute rows equal to valid rows; only tiny/zero-lane cases where padded page count is below CP size need synthetic compute rows.  Physical cache page coverage remains page-granular, but attention/index compute must not ingest dummy tail rows for ordinary long page tails.

Supersedes the earlier 2026-06-05 note that blamed `compute_padding_enabled=False` for the 40387 case.  That older diagnosis matched one intermediate failure mode but no longer matches the current code or the user-confirmed regression boundary.

## 2026-06-05 post long-tail compute-padding fix: still garbage output

User reports the latest run still produces meaningless output after the long-tail compute-padding correction.

Facts already checked and should not be re-opened blindly:

- Remote `/mnt/beegfs/cjy/sglang-dev` hashes matched local for `utils.py`, `nsa_indexer.py`, `nsa_backend.py`, and `hiradix_cache.py` before the latest run.
- Latest log `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260604_225727.log` is after the sync and contains the 40k request.
- For `extend_len` around 40386/40387/40391, `build_batch_page_aligned_in_seq_split_plan()` now reports `compute_padding_enabled=False` and local valid rows equal local compute rows on rank0.
- Latest bad request has `#cached-token: 0`, so HiCache L2->L1 load-back/prefix reuse is not the primary boundary for the first bad output.
- `Prefill batch #new-token=40448` is scheduler page-rounded budget/log accounting, not proof that model compute consumed 40448 rows.
- `HiCache-write owned_positions=5056` is page-owner backup/reservation coverage, not proof that attention/index compute consumed 5056 rows.

Current narrowed suspect set:

1. no-cache current-only CP shared-KV MLA materialize path;
2. no-cache current-only CP shared-KV index/MQA materialize path;
3. CP shared-KV direct-write local loc construction still returning physical/padded rows at a runtime callsite;
4. EAGLE/draft reads malformed prefill KV after target current-only materialize.

Next action: inspect `nsa_backend.py` and `nsa_indexer.py` current-only callsites and add one-shot layer-0 diagnostics if static inspection cannot prove the row contracts.


## 2026-06-05 latest repro: still garbage after long-tail and current-slot fixes

User reports output is still meaningless. Keep the confirmed regression boundary: before bs>1 CP shared-KV changes, BF16 and FP8 E4M3 were both correct.

New facts from code audit to avoid repeated searches:

- Because bs>1 work now attaches a CP shared-KV batch plan even for single-request prefill, several bs=1 callsites route through batch-plan logic. In particular, `NSAIndexer._get_topk_in_seq_cp_pair()` enters `_get_topk_in_seq_cp_pair_batch()` when `get_cp_shared_kv_batch_plan(forward_batch) is not None`, even if metadata `batch_size == 1`.
- FP8 prefill with CP forces NSA implementation to `flashmla_sparse` and topk transform method to `RAGGED`. With `SGLANG_NSA_FUSE_TOPK=1`, `forward_extend()` uses `topk_indices` directly as `page_table_1`; there is no later `topk_indices_offset` repair. Therefore any batch topk coordinate-space mismatch can directly corrupt attention.
- For no-cache current-only prefill, `forward_mla.forward_absorb_prepare()` may rebuild full current `k_nope/k_pe` after CP direct-write because `should_reuse_current_extend_kv(forward_batch)` is true. Attention then consumes the rebuilt full-current KV tensor, so the topk/index coordinate contract must match that rebuilt tensor layout exactly.
- Existing unit coverage checks batch descriptor shapes/lengths, but does not yet prove bs=1 scalar-path and batch-plan-path CP all-gather/rerange outputs are identical, nor that batch topk transform emits indices in the same coordinate space consumed by `flashmla_sparse`.

Next focused audit: prove or disprove that `cp_all_gather_rerange_output()` batch-plan path is bitwise-equivalent to the old scalar CP rerange for bs=1. This is high-signal because both MLA current reuse and index current reuse depend on the rebuilt current tensor and it would explain BF16+FP8 garbage without cache hit.


## 2026-06-05 RED: batch cp_index RAGGED topk loses request-coordinate offset

New focused unit test on remote `g0034` fails before the fix:

```text
test_indexer_ragged_cp_index_batch_uses_request_ragged_offsets
AssertionError: topk_indices_offset_override is None
```

Root-cause evidence:

- `fast_topk_transform_ragged_fused` returns topk indices as local indices plus `topk_indices_offset`; `row_starts` only selects the score window.
- The bs>1 cp_index path compacts each CP segment's K rows into a temporary MQA buffer and sets `row_starts/ks` to compact K bases.
- However attention consumes normal ragged KV coordinates (per-request flattened KV), not the temporary compact segment-coordinate space.
- The current cp_index path calls `metadata.topk_transform(..., cu_seqlens_q=actual_seq_q, batch_idx_list=...)` without `topk_indices_offset_override`, so RAGGED topk derives offsets from compact query cu-seqlens. That is neither request KV base nor compact K base.
- This path is reached for bs=1 after bs>1 metadata attaches `batch_plan`, so it can corrupt the first no-cache prefill and explains garbage output/low accept len without requiring a cache hit.

Planned minimal fix: build a per-compact-query `topk_indices_offset_override` from cumulative `seq_lens_cpu` request KV bases and pass it to all cp_index RAGGED `topk_transform` calls. Do not disable current reuse or batch planning.


## 2026-06-05 GREEN: cp_index RAGGED topk offset fixed

Implemented fix:

- In `_get_topk_ragged_with_cp(cp_index=...)`, compute per-compact-query ragged topk offsets from cumulative `forward_batch.seq_lens_cpu` request KV bases.
- Pass `topk_indices_offset_override` to all cp_index `metadata.topk_transform()` calls: TAI shared batch, TAI current batch, and Python fallback.
- This keeps `row_starts/ks` in temporary compact-MQA-buffer coordinates while final topk indices stay in the normal ragged KV coordinate space consumed by `flashmla_sparse`.

Verification:

- RED test failed before fix on remote: `topk_indices_offset_override is None`.
- After fix, remote `g0034` passed:
  - targeted RED test: `1 passed`;
  - related indexer cp_index tests: `6 passed`;
  - full `test_nsa_cp_utils.py`: `91 passed`;
  - full `test_cp_shared_kv_runtime.py`: `111 passed, 2 subtests passed`.

Remaining runtime validation:

- Need user restart/run real traffic to confirm output text and EAGLE accept length recover. This fix addresses a concrete coordinate corruption in the bs>1-era path that also affected bs=1; it does not prove all true bs>1 cases are complete.

## 2026-06-05 multi-request hang after topk-offset fix

User reports: sending several requests at once can hang. Treat this as a new failure mode after the output-quality fix; do not assume it is the same topk coordinate bug unless logs prove it.

Facts inherited from the previous inspection:

- Latest observed prefill log still showed `#new-seq: 1` only; the bs>1 scheduler gate was enabled but the visible trace had not actually formed a multi-request prefill batch yet.
- Prefill can remain with `#inflight-req: 1` while repeating 64-token health/warmup batches, which points to a disaggregated KV transfer lifecycle stuck after prefill compute, not necessarily active GPU compute.
- Decode configuration in the inspected run had `tp_size=16`, `dp_size=16`, `attn_cp_size=1`, `enable_dp_attention=True`, so `attn_tp_cpu_group` is effectively size 1. Therefore decode-side `poll_and_all_reduce(..., attn_tp_cpu_group)` is not the direct collective hang cause for that configuration.
- Prefill showed Mooncake bootstrap failures/timeouts for some rids and decode showed detokenizer watchdog pending rids. This narrows the next audit to prefill `send_kv_chunk` / inflight polling and decode prealloc/transfer metadata readiness.

High-signal suspects to validate next:

1. `PrefillBootstrapQueue.send_kv_chunk()` can return without sending when `page_indices` is empty. If the caller has already appended the req to `disagg_prefill_inflight_queue`, this creates a request that can never complete transfer polling.
2. `DecodeTransferQueue._commit_transfer_to_req()` treats `metadata.bootstrap_room == 0` after transfer success as "not ready yet". If metadata never arrives, decode can retain a transferred request indefinitely instead of failing with an explicit corruption/ordering error.
3. EAGLE metadata buffer slots are intentionally held until `process_prebuilt()` consumes them; if a multi-request path stores references beyond that point, slot reuse can still corrupt later decode state. This is lower priority until logs show transferred requests reaching prebuilt decode.

Next rule: add bounded fail-fast/diagnostic evidence at the disagg transfer state-machine boundaries before changing scheduling policy or disabling bs>1.

### 2026-06-05 remote deploy mismatch found

Remote active tree on `g0034:/mnt/beegfs/cjy/sglang-dev` differs from local HEAD for `python/sglang/srt/disaggregation/mooncake/conn.py`.

Diff summary:

- Local HEAD imports and calls `validate_transfer_page_count_or_raise(...)` before Mooncake KV transfer.
- Remote still has the old workaround that silently truncates `kv_chunk.prefill_kv_indices` to `len(chunked_dst_kv_indice)` when destination pages are fewer.

Implication:

- The active run can still silently drop CP shared-KV pages during transfer. This invalidates any conclusion drawn from that run about the latest local fail-fast transfer contract.
- This is directly relevant to multi-request hangs/corruption because a mismatched page count can make prefill/decode disagree about transferred KV pages while the transfer layer proceeds instead of failing early.

Next action before further ETE conclusions: sync `mooncake/conn.py` (and any other changed transfer files if hashes differ) to remote and restart before testing. Do not keep debugging latest-run behavior as if the fail-fast transfer validation were active.

### 2026-06-05 fix: final empty KV chunk must still send aux metadata

Code audit found a real disaggregated-transfer lifecycle bug independent of the remote sync mismatch:

- `send_kv_chunk()` wrote final metadata via `disagg_metadata_buffers.set_buf(req)` and then returned early when `page_indices` was empty.
- This is valid for non-final chunk sends because partial tail pages should be delayed.
- It is invalid for the final chunk: if previous chunk sends already transferred all origin KV pages, the final step may have zero new KV pages but still must call `req.disagg_kv_sender.send(empty_pages, state_indices)` so Mooncake queues the final aux metadata transfer and emits terminal status.
- Without that final send, decode can wait for metadata/status while prefill has already put the request into the inflight transfer lifecycle.

Fix:

- Non-final empty chunks still skip.
- Final empty chunks now log `[CP_SHARED_KV_TRANSFER][final_empty_chunk]` and call sender.send with an empty int32 page array so aux metadata/status can complete.

Verification:

- Added `test_prefill_final_empty_kv_chunk_still_sends_aux_metadata`.
- Remote `g0034` container passed targeted test and full file:
  `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_req_to_token_pool.py` → `7 passed`.
- Remote `py_compile` passed for `prefill.py`, `mooncake/conn.py`, and the test file.

Runtime note:

- Code has been synced to `g0034:/mnt/beegfs/cjy/sglang-dev`, but the currently running prefill process was started before this sync. It must be restarted before ETE can validate this fix.

### 2026-06-05 decode death after burst: detokenizer watchdog, not prefill fail-fast

User reports decode died after the transfer lifecycle fix was synced. Current evidence from the latest inspected run:

- Prefill process on `g0034` stayed alive.
- Decode process on `g0035/g0036` was gone after the failure.
- Latest decode log `decode0_20260605_001131.log` showed:
  - a first long request completed with HTTP 200 and normal decode batches;
  - then a burst of multiple dispatched chat requests;
  - then detokenizer health failure: `Server couldn't get a response from detokenizer for last 20 seconds`, followed by `30 requests still pending`;
  - shutdown hit `AttributeError: 'ReqState' object has no attribute 'response_sent_to_client_ts'`.
- Latest prefill log around the same time mainly showed `Prefill bootstrap failed ... Aborted by AbortReq`, which is consistent with decode aborting pending requests during watchdog shutdown. No prior CP shared-KV page-count fail-fast was seen in the inspected grep.

Implication:

- The `response_sent_to_client_ts` exception is a shutdown-cleanup bug, but likely not the primary death cause.
- The primary failure is currently narrowed to decode-side pending request / detokenizer heartbeat / disaggregated transfer lifecycle under burst load.
- Next audit should inspect `TokenizerManager` pending request lifecycle and decode prealloc/transfer queues before changing CP scheduling or transfer contracts again.

### 2026-06-05 root cause: decode health probe treats disagg queue wait as idle

Code audit found a concrete decode-side watchdog false-positive:

- `Scheduler.is_fully_idle(for_health_check=True)` intentionally ignored disaggregation prealloc/transfer queues.
- Therefore, when decode had no running batch but many requests waiting in decode prealloc/transfer, `/health` believed the scheduler was idle and injected a `HEALTH_CHECK_*` generation request instead of returning a scheduler-alive signal.
- That health request could sit behind real disaggregated KV work and produce no detokenizer output within the 20s health timeout.
- The HTTP server then marked the server unhealthy and triggered graceful shutdown, which caused the observed prefill-side `Aborted by AbortReq` bootstrap failures.
- RED tests on remote confirm current behavior:
  - decode prealloc queue still reports idle for health;
  - busy health check is deferred instead of immediately sending `HealthCheckOutput`.

Fix direction:

- Treat disaggregated prealloc/transfer/bootstrap/inflight queues as busy for health checks.
- When a health check arrives while busy, send `HealthCheckOutput` immediately from `process_input_requests`; do not defer it until `process_batch_result`, because a disagg queue wait may have no batch result to piggyback on.
- Keep the shutdown `ReqState.response_sent_to_client_ts` issue as secondary cleanup; it is not the root cause of decode death.

Implemented:

- `process_input_requests()` now sends `HealthCheckOutput` immediately for busy health probes.
- `is_fully_idle(for_health_check=True)` now counts disaggregated prefill bootstrap/inflight queues and decode prealloc/transfer queues as busy.
- `_finish_all_pending_requests_on_shutdown()` now checks `ReqState.time_stats.response_sent_to_client_time` instead of assuming a nonexistent `response_sent_to_client_ts` field, so a future watchdog shutdown can finish pending requests cleanly rather than crashing during cleanup.

Verification:

- Remote RED before fix: health tests failed because decode prealloc queue reported idle and busy health check did not send an immediate signal.
- Remote GREEN after fix:
  `PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_scheduler_health_check.py` → `3 passed`.
- Remote `py_compile` passed for `scheduler.py`, `tokenizer_manager.py`, and the new test.
- Hashes match local and `g0034:/mnt/beegfs/cjy/sglang-dev` for the synced files.

## 2026-06-05 decode 再次退出：当前证据

本轮用户反馈 decode 进程再次死亡后，检查了 g0035/g0036/g0034 最新日志：

- g0035/g0036 上没有存活的 `sglang.launch_server` / `python -m sglang` decode 进程。
- 最新 decode 日志为 `/mnt/beegfs/cjy/log/decode0_20260605_003305.log`，文件停在 00:39:54，最后是大量 `POST /v1/chat/completions HTTP/1.1" 200 OK`，没有 `Traceback`、`Health check failed`、`SIGQUIT`、`RuntimeError`、`NCCL` 或 `CUDA` fatal 行。
- `dmesg` 与 `docker events` 在 00:38-00:41 窗口没有显示容器 OOM / kill / Xid 证据。
- g0036 没有对应的最新 `decode1_20260605_*` 文件；如果 node_rank=1 的 stdout 没有 tee 到文件，则当前固定日志无法覆盖远端第二节点的首个 fatal。
- 最新 prefill 日志 `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260605_003303.log` 在 00:40:04 大量出现 `Prefill bootstrap failed ... KVTransferError(...): Aborted by AbortReq.` 和 `[OUTPUT_ZERO_DEBUG]`。这说明 prefill 侧收到 abort；在没有更早 prefill traceback 的情况下，它更像 decode/router/client 侧中断后的结果，而不是 prefill 首因。
- 之前用户贴出的 decode 启动失败 `ImportError: cannot import name 'validate_transfer_page_count_or_raise'` 是代码同步不一致的明确证据；当前 g0035 容器内 `/sgl-workspace/sglang-tai` 的 `disaggregation/utils.py` 与本地 hash 已一致，并包含该符号。

当前结论：这次不是上一轮已修的 health-check false-positive（至少最新 decode log 没有 health failure 证据）。目前缺失的是 decode node_rank=1 / 子进程首个 fatal 行；不能再基于 prefill `AbortReq` 反推 prefill 是根因。

下一步需要补强的代码侧证据：

1. decode/prealloc/transfer 队列 abort 时应打印 request/room/queue/receiver 状态，避免只在 prefill 看到 `Aborted by AbortReq`。
2. 多节点 decode 启动必须保证 node_rank=1 也有文件日志，或在 rank0 汇总子节点失败原因。
3. 如果再次无 traceback 退出，需要优先找 node_rank=1 stdout / shell 退出码，而不是继续在 prefill HiCache 写入路径重复排查。

## 2026-06-05 SSH 恢复后的复核

SSH 恢复后重新检查远端状态：

- g0034/g0033/g0035/g0036 当前存活的是另一套 `--port 30000 --disaggregation-mode prefill` 进程，不是本轮 cjy 的 `50001` prefill/decode 进程。
- cjy 最新日志仍是：
  - prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260605_003303.log`
  - decode: `/mnt/beegfs/cjy/log/decode0_20260605_003305.log`
- `decode0_20260605_003305.log` 结尾仍停在 00:39:54 的 HTTP 200，没有 fatal traceback。
- `sglang_cp_hicache_20260605_003303.log` 00:40:04 的大量 `KVTransferError(...): Aborted by AbortReq` 是 prefill 侧收到 abort 后的表现；该日志本身仍不能证明 prefill 是首因。
- `decode0_20260605_001131.log` 的旧失败是 health watchdog + `ReqState.response_sent_to_client_ts` 清理异常，已经由本地 `scheduler.py` / `tokenizer_manager.py` 修复覆盖；不能把这个旧日志继续当作当前版本的新根因。

新增诊断：

- `Scheduler.abort_request()` 在 disaggregation 模式下增加 `[DISAGG_ABORT_TRACE]` warning 汇总，打印 abort rid/abort_all、waiting/running、prefill bootstrap/inflight、decode prealloc/transfer 队列长度和 abort_message。
- 目的：下一次复现时能在 decode/prefill scheduler 自己的日志中看到 abort 进入点和队列状态，不再只能从 prefill Mooncake 的 `Aborted by AbortReq` 反推。
- 该日志只在 disaggregation 模式且存在 disagg 队列/abort_all 时触发，不在普通每 tick 热路径触发。

## 2026-06-05 EAGLE metadata slot 生命周期复核

代码审计发现一个比继续追 prefill `AbortReq` 更直接的本地风险点：decode 端 EAGLE 初始 draft metadata 的 pinned CPU buffer slot 被过早释放。

路径：

1. `DecodeTransferQueue._commit_transfer_to_req()` 从 `metadata_buffers.get_buf(idx)` 取出 `output_topk_p` / `output_topk_index` / `hidden_states_tensor` 等 CPU buffer view，挂到 `req` 上。
2. `SchedulerDisaggregationDecodeMixin.get_new_prebuilt_batch()` 调用 `new_batch.process_prebuilt(...)`，把这些 metadata 纳入 prebuilt decode batch。
3. 旧逻辑在 `finally` 中立刻 `_free_decode_metadata_index_if_held(req)`。
4. 但 `process_prebuilt()` 不是最终消费点；真正的 EAGLE draft state 还要等后续一次真实 decode forward/H2D copy 才算安全消费完成。
5. burst 请求下，slot 早释放后可被下一批 transfer 复用并覆盖，导致 decode 初始 draft state 错乱，表现可对应 accept len 降低、输出乱码或后续请求卡死。

修复策略：

- 不 clone 大 tensor；保持原来的零拷贝 CPU metadata view 模型。
- `get_new_prebuilt_batch()` 成功路径不再释放 metadata slot。
- `process_prebuilt()` 抛异常时立即释放，避免错误路径泄漏。
- synthetic prebuilt 步骤内已经结束的请求在 `process_batch_result_prebuilt()` 释放。
- 普通请求在第一轮真实 `process_batch_result_decode()` 中等待 `result.copy_done.synchronize()` 后释放，保证初始 draft metadata 至少存活到第一轮 decode consume 完成。

风险/边界：

- metadata slot 会比旧逻辑多持有一个 decode iteration；这是正确生命周期成本，不是 clone 成本。需要 allocator 容量覆盖 burst prebuilt 请求数。
- 当前修复只证明 slot 生命周期不再早释放；如果下一次仍出现 burst hang，需要结合新增 `[DISAGG_ABORT_TRACE]` 与 `SGLANG_EAGLE_ACCEPT_DEBUG=1` 对比 `metadata_set -> metadata_get -> prebuilt_state`。

验证：

- 本地 `py_compile` 通过：`decode.py`、`scheduler_output_processor_mixin.py`、`test_decode_queue_compaction.py`。
- 远端 g0034 容器 targeted tests 通过：
  `PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/managers/test_scheduler_health_check.py` → `17 passed`。

## 2026-06-06 full GSM8K prefill health-fail investigation

Evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_074842.log` during official GSM8K full run:

- Official 200-question run passed before this: `Accuracy=0.970`, `Invalid=0.000`, no CP fail-fast/fallback/traceback in prefill log.
- Full run reached about 87% (`1150/1319`) before service failure.
- Prefill did not throw a Python exception. First prefill-side terminal symptom was health check:
  - `08:13:32 Health check failed. Server couldn't get a response from detokenizer... last_heartbeat time: 08:12:18`
  - `08:13:35 ... 128 requests still pending ... aborted 128 requests with 503`
- Last normal prefill batch was at `08:12:18`:
  - `#new-seq: 7`, `#new-token: 832`, `#cached-token: 4480`, `#queue-req: 104`, `#inflight-req: 9`.
  - write-through CP backups for node IDs `6271..6277` reached `final ack` and all write locks were released with `remaining_ongoing=0`.
- Decode errors begin after prefill health failure:
  - decode logs start reporting `Could not fetch prefill parallel info from g0034:18992` at `08:13:48`, so decode failure is downstream of prefill bootstrap shutdown, not the first cause.
- Process state after shutdown: no active prefill process/port `17100`; only zombie python children remained. No container OOM evidence was found.

Current hypothesis:

- The scheduler likely stopped progressing after the last normal batch, before producing more detokenizer output.
- Two candidate blocking sites in prefill event loop both use `poll_and_all_reduce_attn_cp_tp_group(...)` and currently lack timing/status logs:
  1. `PrefillBootstrapQueue.pop_bootstrapped()` polls bootstrap queue.
  2. `SchedulerDisaggregationPrefillMixin.process_disagg_prefill_inflight_queue()` polls inflight KV transfer queue.
- Existing logs cannot distinguish whether the stall is in `recv_requests`, `pop_bootstrapped`, `get_next_disagg_prefill_batch_to_run`, `run_batch/process_batch_result`, or `process_disagg_prefill_inflight_queue`.

Next diagnostic requirement:

- Add low-frequency / env-gated timing logs around the prefill event-loop boundaries and around bootstrap/inflight poll-all-reduce. Log only when elapsed time exceeds a threshold or `SGLANG_CP_SHARED_KV_BS_GT1_DEBUG=1` is set.
- Do not change scheduling or transfer semantics until the blocking boundary is confirmed.

Implemented diagnostic:

- `prefill.py` now has env-gated `[CP_SHARED_KV_BS_GT1_TIMING]` markers/timings around:
  - normal prefill event loop: `recv_requests`, `pop_bootstrapped`, `get_next_disagg_prefill_batch_to_run`, `run_batch`, `process_batch_result`, `process_disagg_prefill_inflight_queue`;
  - bootstrap queue poll: `PrefillBootstrapQueue.pop_bootstrapped()` around `poll_and_all_reduce_attn_cp_tp_group`;
  - inflight transfer poll: `process_disagg_prefill_inflight_queue()` around `poll_and_all_reduce_attn_cp_tp_group`.
- The diagnostic is off by default and should not change scheduling or transfer semantics.
- For the next reproduction, use:
  - `SGLANG_CP_SHARED_KV_BS_GT1_TIMING=1`
  - `SGLANG_CP_SHARED_KV_BS_GT1_TIMING_LIMIT=0` so the full GSM8K run does not exhaust the per-event log cap before the failure point;
  - `SGLANG_CP_SHARED_KV_BS_GT1_TIMING_SLOW_MS=0` for the first reproduction so `*_start` markers are available. The last `*_start` without matching `*_done` is the blocking boundary.

## 2026-06-06 restart with timing: current evidence

Remote log: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_124320.log`.

Observed state:

- Prefill process is still alive after the new traffic batch; no current Python traceback or scheduler fatal.
- No `Health check failed` in this run so far.
- No `CP_SHARED_KV_FAIL_FAST`, no `RuntimeError`, no `Scheduler hit an exception`.
- Timing env is active, but the current run uses `SGLANG_CP_SHARED_KV_BS_GT1_TIMING_LIMIT=512` and `SGLANG_CP_SHARED_KV_BS_GT1_TIMING_SLOW_MS=1`; many timing events have already hit the per-rank cap (`4096` total across 8 ranks). If the service later hangs, the boundary marker may be exhausted and not show the final blocking point. For hang reproduction, use `SGLANG_CP_SHARED_KV_BS_GT1_TIMING_LIMIT=0`.

Interesting non-fatal findings:

- `CP_SHARED_KV_FALLBACK` appears 64 times, all seen as `tai_ipc_materialize reason=paged_start_slot_nonzero`; this means some current/prefix materialize paths still fall back from TAI IPC to local materialize plus collective when page-slot range starts from a nonzero slot. This is performance-relevant, not the fatal cause in this run.
- One request abort was observed at 13:00:28 across all CP ranks:
  - `rid=76658e8841dc4db394a4df3052419e56`
  - `prefill_bootstrap=4`, `prefill_inflight=2`, `abort_message=None`
  - The scheduler continued normally afterward, so this abort is not the fatal condition.
- Batch size distribution from CP0 in the parsed window shows bs>1 is active:
  - `bs=1: 411`, `bs=2: 132`, `bs=3: 128`, `bs=4: 62`, `bs=5: 22`, `bs=6: 8`, `bs=7: 4`.

Timing hotspots:

- `event_loop_run_batch_done`: avg ~1031 ms, p95 ~2025 ms, max ~5690 ms. This includes GPU forward and expected model work.
- `scheduler_prefill_prepare`: avg ~73 ms, p95 ~318 ms, max ~920 ms.
- Slowest `scheduler_prefill_prepare` cases are dominated by `prepare_for_extend`, not by `scan_waiting_queue`:
  - example: `elapsed_ms=920.309`, `bs=3`, `extend_lens=[270, 2186, 1113]`, `prefix_lens=[51776, 79296, 94464]`, `prepare_for_extend=870.576 ms`.
  - another example: `bs=1`, `extend_len=17363`, `prefix_len=137728`, `hicache_ready_to_load=43 ms`, `prepare_for_extend=716 ms`.
- `scan_waiting_queue` can still reach about 100 ms when queue is nontrivial, but it is not the largest hotspot in these slow samples.

Current conclusion:

- This restart has not reproduced the previous prefill-death symptom yet.
- The strongest evidence from this run is performance/control-path overhead in `prepare_for_extend` under large cached-prefix / bs>1 cases, plus residual TAI IPC fallback for nonzero page-slot materialize.
- For a future fatal reproduction, the current timing cap is insufficient; restart with unlimited timing limit before relying on the last `*_start`/`*_done` boundary.

## 2026-06-06 full GSM8K accuracy verification

Command run on g0034 container (`/sgl-workspace/sglang-tai`) against router `g0034:17100`:

```bash
OPENAI_API_KEY=EMPTY PYTHONPATH=python python benchmark/gsm8k/bench_sglang.py \
  --host g0034 --port 17100 --backend srt \
  --data-path /mnt/beegfs/cjy/data/gsm8k_test.jsonl \
  --num-questions 1319 --parallel 64 \
  --temperature 0.0 --top-p 1.0 --max-new-tokens 512 \
  --result-file /mnt/beegfs/cjy/gsm8k_bench_sglang_full_20260606_132651.jsonl \
  --raw-result-file /mnt/beegfs/cjy/gsm8k_bench_sglang_full_raw_20260606_132651.jsonl
```

Result evidence:

- Progress reached `1319/1319`.
- `Accuracy: 0.955`.
- `Invalid: 0.000`.
- `Latency: 311.296 s`.
- `Output throughput: 451.072 token/s`.
- Summary file: `/mnt/beegfs/cjy/gsm8k_bench_sglang_full_20260606_132651.jsonl`.
- Raw file: `/mnt/beegfs/cjy/gsm8k_bench_sglang_full_raw_20260606_132651.jsonl`.
- Log file: `/mnt/beegfs/cjy/gsm8k_bench_sglang_full_20260606_132651.log`.
- Post-run router health: `http://127.0.0.1:17100/health` returned `OK`.
- Post-run latest prefill log `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_124320.log` had no matches for:
  `Health check failed`, `Traceback`, `RuntimeError`, `Scheduler hit an exception`, `CP_SHARED_KV_FAIL_FAST`, `KVTransferError`, `SIGQUIT`.

Interpretation:

- Full GSM8K correctness is back to the expected official number (`0.955`) under current bs>1 CP shared-KV/HiCache configuration.
- The previous full-run prefill death was not reproduced in this run.
- This only verifies GSM8K accuracy and basic post-run service health; it does not close remaining performance issues such as `prepare_for_extend` CPU overhead or `tai_ipc_materialize reason=paged_start_slot_nonzero` fallback.

Caveat:

- `wc -l` on the raw result file reported `1318`, while the benchmark progress reached `1319/1319` and the summary result records `num_requests=1319`. Treat the printed benchmark summary as the accuracy source of truth unless raw-record completeness is needed for per-question diagnosis.

## 2026-06-06 second full GSM8K cache-check prefill death

Remote run under test:

```bash
OPENAI_API_KEY=EMPTY PYTHONPATH=python python benchmark/gsm8k/bench_sglang.py \
  --host g0034 --port 17100 --backend srt \
  --data-path /mnt/beegfs/cjy/data/gsm8k_test.jsonl \
  --num-questions 1319 --parallel 64 \
  --temperature 0.0 --top-p 1.0 --max-new-tokens 512 \
  --result-file /mnt/beegfs/cjy/gsm8k_bench_sglang_full_cachecheck_20260606_135051.jsonl \
  --raw-result-file /mnt/beegfs/cjy/gsm8k_bench_sglang_full_cachecheck_raw_20260606_135051.jsonl
```

Observed failure:

- Router health was `OK` before starting the second run.
- The benchmark progressed to about `77%` and then stalled; the benchmark process had to be killed manually.
- Prefill port `50001` was gone after the stall while router port `17100` was still listening.
- Latest prefill log remained `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_124320.log`.
- No prefill-side Python traceback or explicit CP fail-fast was found immediately before death.
- Last normal prefill batch before watchdog was at `13:54:35` with `#new-seq: 7`, `#new-token: 448`, `#cached-token: 4928`, `#queue-req: 42`, `#inflight-req: 12`.
- Watchdog symptom:
  - `13:55:17 Health check failed. Server couldn't get a response from detokenizer ... last_heartbeat time: 13:54:35`.
  - `13:55:20 ... 65 requests still pending ... aborted 65 requests with 503`.
- Last timing evidence before death shows bootstrap queue did not fully drain:
  - `bootstrap_poll_done elapsed_ms≈1.0 queue=7 polls_head=[2,2,2,2,2,2,1]` on multiple ranks.
  - `event_loop_pop_bootstrapped_done ... bootstrapped=6 ... bootstrap_remaining=1`.
- Right after that, CP write locks for nodes `9803..9809` were released, followed by a burst of HiCache `load_back CP` logs for small host hits. These load_back entries report `SUCCESS`, so this is evidence of activity near the stall, not yet proof of a failing load_back transfer.
- Stale rid to investigate: `76658e8841dc4db394a4df3052419e56` was aborted much earlier at `13:00:28` with `prefill_bootstrap=4 prefill_inflight=2`, but appears again in the final watchdog `remaining_rids` list at `13:55:20`.

Current investigation direction:

- Root cause is not established yet.
- Two high-priority code paths to inspect:
  1. `PrefillBootstrapQueue.pop_bootstrapped()` ordering/cleanup behavior when one item remains pending while later items may be ready.
  2. Abort cleanup for requests already present in prefill bootstrap/inflight queues and tokenizer pending state.
- Do not treat the final HiCache load_back burst as root cause until a missing `*_done` timing boundary or code path proves it blocks scheduler progress.

### 2026-06-06 root-cause finding: L1 owner-lane free-room was only diagnostic on success paths

Evidence added after code inspection:

- Final cache-check failure ended after a burst of CP HiCache load-back for tiny host hits:
  - `host_hit_len=64` appeared `104` times in the final two-second window across CP ranks;
  - `host_hit_len=128` appeared `8` times;
  - owner-lane availability in those logs fell to `available_by_owner[0]=1` while `free_room_deficit_by_owner[0]≈443`.
- `hiradix_cache.load_back()` logged `free_room_deficit_by_owner`, but only evicted on `deficit_by_owner` (exact required pages), not on free-room deficit. Therefore successful tiny load-backs could drain owner lane 0 without replenishing L1 free room.
- `alloc_paged_token_slots_extend()` only called `_evict_for_compute_owner_lanes()` after the exact owner-lane allocation failed. Since `alloc_extend_compute_owner()` itself only enforces exact required pages, L1 free-room ratios were not maintained on successful extend allocations either.
- This explains why the second GSM8K run, unlike the first, can regress after cache is populated: many short cached prefixes map to the first CP owner lane, consume owner-0 pages, and leave the scheduler with little L1 room for subsequent cache-hit/extend work.

Fix direction:

- Preserve exact fail-fast semantics.
- Add proactive owner-lane free-room eviction before successful owner-aware extend allocation when L1 free-room trigger/target ratios are enabled.
- Add best-effort CP HiCache load-back owner-lane free-room eviction when exact capacity is sufficient but free room is below trigger. Failure to satisfy the free-room target should not reject the load-back; exact deficits still remain strict.

### 2026-06-06 L1 owner-lane free-room fix implemented

Implementation:

- `alloc_paged_token_slots_extend()` now proactively calls CP owner-lane eviction before the first successful owner-aware allocation when L1 free-room target/trigger ratios are enabled. This keeps the configured free room from being only a failure-path mechanism.
- CP HiCache `load_back()` now performs best-effort owner-lane free-room eviction when exact `deficit_by_owner` is zero but `free_room_deficit_by_owner` is positive. Exact deficits still remain strict; free-room misses do not reject load-back.

Regression coverage:

- Added `test_compute_owner_alloc_maintains_l1_free_room_before_successful_alloc` to lock the case where exact allocation succeeds but lane free-room trigger requires proactive eviction.
- Remote g0034 container verification:
  - `PYTHONPATH=python python -m py_compile python/sglang/srt/mem_cache/common.py python/sglang/srt/mem_cache/hiradix_cache.py test/registered/unit/mem_cache/test_cp_shared_kv_layout.py` passed.
  - Targeted owner-lane tests: `3 passed`.
  - Full `test_cp_shared_kv_layout.py`: `38 passed`.

Open runtime validation:

- No new prefill process has been started from this fix yet. The currently latest remote log remains the old failed run (`sglang_cp_hicache_20260606_124320.log`) with prefill port `50001` absent and router port `17100` still listening.

### 2026-06-06 correction: free-room trigger/target semantics

User correction applied to both L1 owner-lane and L2 host admission semantics:

- Trigger condition must account for the pending allocation/reservation:
  - trigger when `available < required + trigger_room`;
  - equivalently, when `available - required < trigger_room`.
- Once triggered, `target_room` is the minimum eviction chunk.  Evict
  `max(required - available, target_room)`.
- Rationale: `required + target_room` leaves a full target watermark after the
  current allocation and can waste too much L1/L2 cache residency under many
  small cache-hit requests.  Evicting only to `max(required, target_room)`
  available is also wrong because it treats `target_room` as an available
  watermark rather than a minimum eviction chunk.

Tests added/updated:

- L1: direct owner-lane helper test now covers `available=2, required=1, trigger=2`, which must trigger because the pending allocation would leave only one page.
- L2: `_free_room_deficit` and CP host admission tests now expect
  `max(required - available, target_room)` after trigger.

## 2026-06-07 repeat GSM8K precision run blocked by prefill death

Remote run attempted for the requested first repeat:

```bash
OPENAI_API_KEY=EMPTY PYTHONPATH=python python benchmark/gsm8k/bench_sglang.py \
  --host g0034 --port 17100 --backend srt \
  --data-path /mnt/beegfs/cjy/data/gsm8k_test.jsonl \
  --num-questions 1319 --parallel 64 \
  --temperature 0.0 --top-p 1.0 --max-new-tokens 512 \
  --result-file /mnt/beegfs/cjy/gsm8k_bench_sglang_full_repeat1_20260606_171214.jsonl \
  --raw-result-file /mnt/beegfs/cjy/gsm8k_bench_sglang_full_repeat1_raw_20260606_171214.jsonl
```

Observed failure:

- Benchmark log: `/mnt/beegfs/cjy/gsm8k_bench_sglang_full_repeat1_20260606_171214.log`.
- Progress reached roughly the middle of the dataset, then failed with router/decode error:
  `decode_internal_error: Could not fetch prefill parallel info from g0034:18992 after 20 attempts`.
- After failure, only router port `17100` was listening; prefill port `50001` and bootstrap port `18992` were gone.
- Prefill log: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_164817.log`.
- Last normal prefill batch before watchdog:
  `17:14:54 #new-seq=5 #new-token=640 #cached-token=3200 #queue-req=47 #inflight-req=9`.
- CP write locks for nodes `5822..5826` were released immediately after that batch.
- Watchdog symptom:
  - `17:15:37 Health check failed ... last_heartbeat time: 17:14:54`.
  - `17:15:41 ... 65 requests still pending ... aborted 65 requests with 503`.
- No prefill-side Python traceback, explicit OOM, or CP fail-fast was found immediately before the watchdog shutdown.

Important correction:

- User confirmed this repeat was run on a new prefill process started after the free-room formula corrections. Therefore the failure is valid evidence that the latest free-room correction alone does not fully fix the repeat-GSM8K prefill death.
- The second requested repeat was not started because the first repeat took down prefill/bootstrap.

Current next validation requirement:

- Treat this as an active post-fix hang/regression, not an old-process artifact.
- If this hang needs deeper diagnosis, run with an uncapped or higher CP bs>1 timing log limit so the final scheduler/bootstrap boundary is visible; the current log reaches watchdog without a direct failing traceback.

### 2026-06-07 repeat GSM8K post-fix death: current bounded finding

New evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_164817.log`:

- The process did not die with a Python traceback, explicit OOM, CP fail-fast, or `Scheduler hit an exception`.
- Last normal prefill batch was `17:14:54` with `bs=5`, `new_token=640`, `cached_token=3200`, `queue_req=47`, `inflight_req=9` on all CP ranks.
- The final batch's node/rid write-through backup reached `submit_write_cp_layer final ack`, `attached prepared CP backup`, and `writing_check released 5 write locks` on all CP ranks.
- The final watchdog pending list still contained the final-batch rids (`3732391e...`, `17f4f067...`, `55a9e51c...`, `2827e296...` etc.), so the `POST /generate 200 OK` lines at `17:14:54` are not proof that the final batch completed; they may be earlier inflight completions.
- `GET /health` was healthy through `17:14:18`; the watchdog started at `17:15:17` and reported `last_heartbeat=17:14:54`.
- The timing diagnostics were ineffective at the actual failure point because the run had a malformed env assignment: `SGLANG_CP_SHARED_KV_BS_GT1_TIMING_LIMIT` parsed as `"-1SGLANG_CP_SHARED_KV_BS_GT1_DEBUG=1"` and fell back to default `256`, so common scheduler boundary events hit their per-event cap around `16:59`.

Current inference:

- The strongest current suspicion is a scheduler-side hang after the final batch, before any later detokenizer response. The likely boundaries are still:
  1. `process_disagg_prefill_inflight_queue()` polling inflight KV transfer;
  2. `PrefillBootstrapQueue.pop_bootstrapped()` / subsequent scheduler admission for the next batch;
  3. `stream_output()` / ZMQ send if the detokenizer side stops consuming.
- Existing logs cannot distinguish these because the final boundary markers were capped before the failure.
- Do not keep re-reading this log expecting a hidden traceback; the current log does not contain one.

Diagnostic change added locally:

- Added an env-gated CP shared-KV poll queue consensus guard behind the existing `SGLANG_CP_SHARED_KV_BS_GT1_DEBUG=1`.
- The guard validates `(queue_len, rid_hash)` across attn-TP and attn-CP CPU groups before the old variable-length poll tensor all-reduce. If queue length or rid order diverges, it raises `[CP_SHARED_KV_FAIL_FAST][poll_queue]` instead of risking a silent collective hang.
- This intentionally adds extra scalar collectives only when the existing `SGLANG_CP_SHARED_KV_BS_GT1_DEBUG=1` is enabled; normal production and timing-only paths do not pay this cost.
- Regression test added: `test_poll_consensus_debug_fails_before_shape_mismatch_hang`.

Next reproduction requirements:

- Use correctly separated env assignments:
  - `SGLANG_CP_SHARED_KV_BS_GT1_TIMING=1`
  - `SGLANG_CP_SHARED_KV_BS_GT1_TIMING_LIMIT=0`
  - `SGLANG_CP_SHARED_KV_BS_GT1_TIMING_SLOW_MS=0`
  - optional: `SGLANG_CP_SHARED_KV_BS_GT1_DEBUG=1` for one diagnostic run only.
- If the next run reports `[CP_SHARED_KV_FAIL_FAST][poll_queue]`, root cause is queue length/order divergence across ranks.
- If it still watchdogs without poll fail-fast, the next suspect is a call-count mismatch between ranks or a non-poll blocking site (`stream_output`, admission/eviction planning, or receive loop); use the uncapped boundary markers to locate the last `*_start` without matching `*_done`.

## 2026-06-07 cache-hit GSM8K accuracy regression

New user-provided evidence on `g0034:17100` with full GSM8K (`1319` questions, parallel=64, temp=0):

```text
cold/first run:  Accuracy: 0.955  Invalid: 0.000  Latency: 207.174 s  Output throughput: 668.366 token/s
second run:      Accuracy: 0.687  Invalid: 0.001  Latency: 211.220 s  Output throughput: 660.263 token/s
```

Interpretation:

- The cold path can still reach the expected GLM-5.1 GSM8K accuracy, so the base model, decode EAGLE, FP8 decode, and non-hit prefill path are not globally broken.
- The immediate second run is a cache-hit path and drops accuracy sharply. This shifts the top suspect away from generic EAGLE/FP8 decode and toward cache-hit semantics: radix/HiCache match length, page-tail valid length, CP shared-KV load/materialize, or cached index/MLA reuse under page-aligned storage.
- Throughput remains similar between runs, so this is correctness corruption rather than a service-death or obvious slow-path-only issue.

Current top hypothesis to verify next:

- Some path exposes page-rounded/padded KV as valid cached prefix, or uses cached prefix metadata without flooring to the last valid page boundary. A later request can then skip recomputing tail tokens or consume stale/padded page data, which would be invisible on the cold run and show up only after cache hits.

Immediate code audit targets:

1. `hiradix_cache.py` CP page-floor/prune/match logic: cached match length returned to scheduler must be valid-token/page-floor length, never physical padded extent.
2. `radix_cache.py` / `schedule_batch.py`: `prefix_indices`, `host_hit_length`, `cache_protected_len`, and `extend_input_len` must not include padded tail tokens.
3. `nsa/utils.py` and current reuse paths: batch compute padding and physical page padding must not alter request logical lengths used for top-k/MLA attention.
4. HiCache load path: host backed CP node should obey the same page-minimum contract; if a node is backed only to page floor, cache hit must not round up to the next page.

## 2026-06-07 继续定位：cache-hit 回退集中到 slot-remap 的矩形 page table 合同

新增证据：

- 两次 GSM8K 全量同进程复跑：cold run `Accuracy=0.955`，immediate cache-hit run `Accuracy=0.687`。cold 正确说明模型、decode 基本路径、FP8 非命中路径不是全局损坏；错误集中在 cache-hit 前缀被 prefill 重新消费后。
- 远端日志 `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_180649.log` 第二轮出现稳定 cache-hit batch：`#new-seq: 5`, `#new-token: 320/384`, `#cached-token: 3520/3712`，约等于每请求 704+ cached tokens。
- `hiradix_cache` 初查没有看到把 non-page-aligned hit 向上 round-up 的证据；当前仍是 page-floor 方向。因此下一主线不是 radix round-up，而是 cache-hit prefix materialize/compose。
- `nsa_backend.py` 使用 `metadata.real_page_table` 构造 token/index slot-remap；`real_page_table` 来自矩形 `req_to_token[:, :max_seq_len]` 按 page stride 取样。bs>1 且请求长度不同/slot 复用时，矩形尾部可能包含请求 valid page 之外的旧 loc。
- `cp_shared_kv_runtime.build_slot_page_inverse()` 对 duplicate logical page 使用 `scatter_`，同一 logical page 会映射到最后一次出现的 slot。若 valid prefix/current page 也出现在某个请求的矩形无效尾部，page_inverse 可能把合法 topk loc remap 到一个未被 prefix/current spans materialize/reduce 的 stale slot，cache-hit attention 会读到 0/错误 KV。该路径只在 cache-hit prefix materialize 时高概率触发，符合 cold 正常、hit 掉点的现象。

待验证/修复方向：

1. 给 `metadata.real_page_table`/index logical page table 增加 batch-aware valid-page mask：每个 request 只保留 `ceil(seq_len/page_size)` 或 `ceil((prefix_len+extend_len)/page_size)` 范围内的 page，范围外置 0，不参与 slot inverse。
2. 增加单测：构造 bs>1 矩形 logical_pages，尾部 stale slot 重复了前面 valid logical page，验证 slot remap 不能把 valid loc 映射到 stale slot。
3. 修复应优先在 slot-remap 输入处清洗 invalid rectangular tail，而不是禁止 cache-hit/current reuse。

## 2026-06-07 修复：在 NSA metadata 边界清理 rectangular page-table tail

- 变更点：`NSAMetadata.real_page_table` 构建后立即调用 `mask_batch_logical_pages_to_valid_lengths()`，按每个 request 的有效 `seq_len` 保留 `ceil(seq_len/page_size)` 个 page，后续矩形尾部统一置 0。
- 选择该位置的原因：`real_page_table` 同时喂给 MLA KV materialize、index materialize、prefetcher 与 slot-remap cache。集中清理比在每个 consume/materialize 调用点单独清理更不容易遗漏。
- 保留的合同：最后一个有效 partial page 仍保留；不会把 token 级 valid length 截成 compact token cache，只清理 page-table 的 invalid row tail。
- 覆盖的风险：当 bs>1 不同 seq_len 混排、request slot 复用导致矩形尾部出现旧 logical page，`build_slot_page_inverse`/TAI inverse 的 duplicate page 写入不再能把有效 page remap 到 stale slot。
- 验证：远端容器 `py_compile` 通过；`test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_valid_page_mask_prevents_stale_rectangular_tail_remap` 通过；完整 `test_cp_shared_kv_runtime.py` 通过 `114 passed, 5 warnings, 2 subtests passed`。
- 未验证：最新 ETE cache-hit GSM8K 二次全量精度仍需用户重启新代码后复测。

### 2026-06-07 澄清：cuda graph replay 不是当前 prefill ETE 主线

- 用户确认当前 prefill 不走 cuda graph。
- 因此 `init_forward_metadata_replay_cuda_graph()` 中同类 rectangular-tail 复用风险不作为本次 GSM8K cache-hit 掉点的 active root cause。
- 当前主线收敛到非 cuda-graph `init_forward_metadata()` 构建的 `real_page_table`，以及其下游 MLA/index materialize、prefetch、partial/current reuse。

### 2026-06-07 继续审计：`forward_indexer()` 的 direct req_to_token page-table 路径
- 发现 `nsa_indexer.forward_indexer()` 仍直接从 `forward_batch.req_to_token_pool.req_to_token[:, ::page_size]` 构造 `block_tables`，再传入 `_maybe_materialize_shared_index_buffer()`。
- 这条路径若在 CP shared-KV + cache-hit 下启用，会绕过 `nsa_backend.init_forward_metadata()` 中对 `real_page_table` 的 logical-tail mask，存在同类 stale rectangular-tail 风险。
- 但当前 CUDA/Hopper prefill 的 `forward_cuda()` 在 `_is_cuda or _is_hip` 分支内走 `_get_topk_paged/_get_topk_in_seq_cp_pair/_get_topk_ragged`，只有非 CUDA/HIP 分支才调用 `forward_indexer()`。
- 结论：这是潜在维护风险，不是当前 prefill ETE 的 active root cause；当前继续审计 CUDA active path。

### 2026-06-07 继续审计：CUDA graph replay / MTP precompute 的 direct req_to_token 路径
- 发现 `nsa_backend.init_forward_metadata_replay_cuda_graph()` 以及 `nsa_backend_mtp_precompute.py` 的 decode/target-verify/draft precompute 分支也直接从 `req_to_token` 构造 token/page table，并在 `real_page_size > 1` 时 transform 成 `real_page_table`。
- 这些路径当前未调用 `mask_batch_logical_pages_to_valid_lengths()`，因此在 decode/target-verify/draft CUDA graph replay 或 MTP precompute 场景中存在同类 stale rectangular-tail 风险。
- 用户已确认当前 prefill 不走 cuda graph；因此它们不是本次 prefill GSM8K cache-hit 掉点 active root cause。
- 后续若要覆盖 decode graph / draft graph，应在 precompute/replay 边界同样按 logical `seq_lens` mask real-page-table，且需要专门测试，不能把它混入当前 prefill 修复判断。

### 2026-06-07 再确认：page 物理粒度不等于 token 可见长度向上取整

本轮代码复核结论：

- `ReqToTokenPool.alloc()` 复用 slot 时只从 `free_slots` 取回 `req_pool_idx`，`free()` 也只把 slot 放回队列并清空 `req.req_pool_idx`；`req_to_token` 行内容不会被清零。因此矩形 batch 读取到 `max_seq_len` 时，短请求 valid token 之后的 tail 可以携带旧请求残留。
- `write_cache_indices()` 只写 `[0, prefix_len)` 与 `[prefix_len, seq_len)`，不会清理 `[seq_len, max_seq_len)`。这进一步确认 stale tail 是真实输入状态，不是单测构造出来的假风险。
- CP HiCache 的合同是：radix/scheduler 可见长度使用 `logical_len`/`valid_len`，物理 host/device reservation 使用 `padded_len`，且 `padded_len` 必须是 `page_size` 倍数。也就是说 cache 存储和释放以 page 为最小物理单位，但命中长度和 split 仍不能把 padding token 暴露为有效 token。
- CP HiCache split 强制 `split_len % page_size == 0`；exact non-page-aligned hit 会 floor 到上一页，stale tail child 会被 prune；chunked prefill backup 也只 backup 到 `floor_to_page_len()`。这些都支持同一个合同：可以牺牲一个 sub-page tail 来换 page 管理一致性，但不能把 page padding 当成有效语义 token。
- 因此 `real_page_table` 的 tail mask 是正确方向：保留 `ceil(valid_seq_len / page_size)` 个 page，包括最后一个 partial page；只把每行 valid page 数之后的矩形尾部置 0，避免 stale duplicate page 参与 `page_inverse.scatter_()`。这不会丢失以 page 存储的合法 cache，只会阻断无效矩形尾部覆盖 slot inverse。

继续审计到的类似风险：

1. `metadata.page_table_1` 仍保留 token 粒度矩形表，未整体 mask。当前 CUDA prefill 的 CP shared-KV materialize/slot-remap 已经改用 masked `metadata.real_page_table`，但 `topk_transform()` 的 paged fused path 仍以 `page_table_1` 为输入。现有 debug validator 只检查输出是否在整个 page table 的 min/max 范围内，不检查“输出 loc 是否属于该 request 的 valid prefix”。如果 cache-hit 复测仍掉点，下一步应加一个 env/debug 级 per-row membership validator，而不是继续禁用 current/partial reuse。
2. `nsa_indexer.forward_indexer()` 的非 CUDA/HIP 路径会直接从 `req_to_token[:, ::page_size]` 构造 `block_tables`，绕过 metadata 边界 mask。当前 Hopper CUDA prefill 不走这条路径，归类为维护风险。
3. CUDA graph replay / MTP precompute 路径仍存在 direct `req_to_token` page-table 构造。用户确认 prefill 无 cuda graph，所以不是当前 active root cause；后续若要覆盖 decode/draft graph，应单独补 mask 和测试。
4. MHA FP8 的 `page_table_1_flattened` 是按每行 `indexer_seq_lens_cpu` 做 `page_table[i, :kv_len]` 的 valid slice concat，不读矩形 tail；当前看不是同类 stale-tail 风险。
5. prefill disaggregation transfer 当前读 `req_to_token[req_pool_idx, start:end]` / `[:seq_len]` 这类 valid token slice；这类路径本身不消费矩形 tail。仍需保留已有 page-count fail-fast，但它不是本轮 cache-hit 精度掉点的主嫌。

后续判据：

- 如果最新 ETE 二次 GSM8K 精度恢复，说明主要 root cause 是 `real_page_table` slot-remap tail。
- 如果仍掉点，优先打开/补强 topk per-request membership validator，检查 `topk_indices` 是否引用了每个 request valid range 外或错误 request 的 loc；不要先做大范围 fallback/禁用。

### 2026-06-07 二次 GSM8K 仍掉点：风险从“所有 cache-hit”收敛到 non-page-aligned CP HiCache 命中长度/内容

新远端证据来自 `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_193039.log`：

- 用户同进程连续两轮 GSM8K：第一轮 `Accuracy=0.953`，第二轮 `Accuracy=0.692`，吞吐基本相同。
- 第一轮并不是完全无 cache：从 `19:35:08` 起已经出现大量 `#new-seq: 5, #cached-token: 3200`，即约每请求 640 cached tokens；第一轮仍正确。
- 第二轮主要变成 `#new-token: 320/384, #cached-token: 3520/3584/3648...`，即命中更长、更贴近每个 GSM8K 问题自身的缓存。
- 第一轮开始处 CP HiCache 写入了 `node_id=4 logical_len=698`；这是 non-page-aligned node，物理 page 覆盖会到 704 tokens。后续第二轮的典型 `cached-token=3520` 正好对应 5 个请求各 704 tokens。

当前推断：

- “cache-hit 本身全部坏”不是最强解释，因为第一轮已有 640-token 共享前缀命中且精度正常。
- 更强嫌疑是：某个 CP HiCache/radix/load-back/scheduler-visible 前缀长度路径，把 `logical_len=698` 的 node 以物理 `padded_len=704` 暴露给 scheduler/attention，或者 load-back/backup 的最后 partial page 内容/索引与 logical valid token 不一致。
- 这会让第二轮复用 per-question cache 时跳过 6 个应计算 token，或消费 page padding/stale KV；冷跑和只命中 640 共享前缀时不明显，复跑命中 question-specific non-page-aligned node 时掉点。

下一步审计目标：

1. `hiradix_cache.match_prefix()` / `_match_prefix_helper()` / `load_back()`：确认返回给 scheduler 的 `prefix_indices` 长度来自 logical valid length，而不是 `padded_len`。
2. `schedule_batch.py`：确认 `cache_protected_len`、`extend_input_len`、日志里的 `#cached-token` 是否可能为满足 page 对齐被向上取整。
3. CP HiCache backup/load path：确认 host/device node 的 `value` 只包含 logical indices；物理 page reservation/owned positions 不应改变 node visible length。
4. 若代码确认没有 length exposure，再回到 top-k membership validator；但当前日志优先级应先查 698→704 这条链。

### 2026-06-07 复测后收敛：第二轮掉点不是所有 cache-hit 坏，而是 first-run suffix page 持久化复用可疑

最新证据来自 `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260606_193039.log`，对应用户同进程两轮 GSM8K：第一轮 `Accuracy=0.953`，第二轮 `Accuracy=0.692`。

日志事实：

- 该日志内没有 `CP_SHARED_KV_FAIL_FAST`、`CP_SHARED_KV_FALLBACK`、`RuntimeError`、`Traceback`、`Health check failed`、`HiCache-load`、`CacheCtrl-load`、`load_back`。
- 第一轮已经有稳定 cache-hit：CP0 典型 `#new-seq: 5, #new-token: 640/704/576, #cached-token: 3200`，约等于每请求 640 cached tokens；第一轮仍正确。
- 第二轮主要变成 `#new-seq: 5, #new-token: 320/384, #cached-token: 3520/3584/...`，即每请求多复用了约 64 tokens 的 question-specific suffix page。
- 第一轮 CP0 写入大量 non-page-aligned suffix node，例如 `logical_len=95/81/140/66/86`，且 `owned_positions=64`；bs=5 时 CP0 常见 `valid_local_tokens=320`，即每个请求写入首个 owner page。
- 第二轮 CP0 `local_out_cache_loc` 多为小 suffix：如 `split_tokens=165 out_cache_tokens=168 local_tokens=165`、`split_tokens=63 out_cache_tokens=64 local_tokens=63`，说明当前 extend/write 仍按 valid rows 截掉 physical padding。

修正当前 root-cause 优先级：

- 仅用“radix/scheduler 把 698 向上暴露成 704”解释不充分。代码中 CP exact hit 仍有 page-floor 方向，且日志也能解释为第二轮命中共享 640 page 后，再命中第一轮写入的每题 suffix 首个 64-token page。
- 因此当前更强嫌疑是 bs>1 下持久化 direct write 的 row-order/physical-loc contract：第一轮正常使用 current K/V 参与计算并得到正确答案，但同一轮写入 device/HiCache 的 MLA KV 或 NSA index K 在第二轮作为 prefix 被复用时内容或行序错误。
- 需要优先审计 `get_cp_shared_kv_local_out_cache_loc()`、`select_cp_local_valid_rows_for_cache_write()`、MLA direct store、index direct store 之间的行序一致性，以及 prefix materialize 读取这些 page 时是否按相同 owner-lane order 解释。

下一步最小验证：

1. 构造/补强单测覆盖 bs=5、page_size=64、prefix=640、extend=[95,81,140,66,86] 这类 GSM8K 形态，验证 local valid row selector 输出顺序与 local physical out loc 顺序逐 request/page 对齐。
2. 若单测无法暴露，加入 env-gated、限频的 runtime validator：在 first-run suffix direct write 后读取刚写入的 index/MLA buffer，与 current local rows 做 checksum/row-id 对比。优先验证 index K，因为它有较清晰的 `local_key -> physical_out_loc` 写入路径；MLA FP8 packed store 需要额外确认量化/布局。
3. 不再通过大范围禁用 current/partial reuse 判断根因；否则会掩盖“第一轮 direct write 与第二轮 persistent reuse 不等价”的核心问题。
