Stabilize CP HiCache page-tail ownership under EAGLE reuse
CP shared KV and HiCache now keep page-aligned physical ownership while preserving valid-token radix semantics. Repeated tiny EAGLE exact hits free duplicate tail pages instead of leaking one allocator page, owner-lane load-back uses page-vector admission/eviction, and single-DP idle schedulers avoid entering an unnecessary MLP-sync collective. The commit also records the current page-aligned cache contract and adds gated decode-side EAGLE accept diagnostics so future accept-length collapses can be tied to draft KV/state transfer evidence instead of more prefill cache speculation. Constraint: CP HiCache allocator ownership is page-granular while radix matching remains valid-token based. Constraint: New diagnostics must be gated and must not alter normal EAGLE, transfer, or cache behavior. Rejected: Padding short requests to cp_size or 2*cp_size pages | wastes KV capacity and still hides valid-tail lifecycle bugs. Rejected: Adding more unconditional collectives to prove CP consistency | hot-path collectives previously caused severe performance risk. Confidence: medium Scope-risk: broad Directive: Do not reintroduce silent fallback for CP shared KV/HiCache paths; warning-level fallback or fail-fast is intentional. Tested: git diff --check Tested: local py_compile for all modified Python files Tested: remote g0034 container py_compile for modified Python/test files Tested: remote g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/managers/test_scheduler_dp_attn_mixin.py => 114 passed, 5 warnings, 2 subtests passed Not-tested: full ETE traffic rerun after this commit Not-tested: CUDA/TAI kernel benchmark coverage for all production shapes
This commit is contained in:
@@ -1558,3 +1558,411 @@ Verified C22:
|
||||
`73 passed, 2 subtests passed`.
|
||||
- Local pytest remains blocked by missing optional deps such as `orjson`; remote
|
||||
container is the authoritative verification environment for this path.
|
||||
|
||||
## C23: Repeated EAGLE/bigram requests can expose 65-token tiny suffixes on 64-token pages
|
||||
|
||||
Finding:
|
||||
|
||||
- A repeated request can still show `extend_lens=[65]` even when the logical
|
||||
prompt is identical and page flooring is only supposed to lose at most
|
||||
`page_size - 1` tokens.
|
||||
- The extra token comes from EAGLE/bigram key semantics plus SGLang's existing
|
||||
one-token prefix-cache holdback:
|
||||
- request sequence length in the observed log: `seq_len=40385`;
|
||||
- scheduler prefix match caps token keys at `input_len - 1`;
|
||||
- EAGLE radix keys are bigrams, so `convert_to_bigram_key()` reduces that key
|
||||
length by one more token;
|
||||
- effective matchable key length becomes `40385 - 2 = 40383`;
|
||||
- page-flooring to 64-token pages gives `40320`;
|
||||
- forward therefore sees `40385 - 40320 = 65` extend tokens.
|
||||
|
||||
Implication:
|
||||
|
||||
- For non-EAGLE page-granular reuse, the repeated-request suffix can be up to
|
||||
`1 + (page_size - 1)` tokens because the last token is held back.
|
||||
- For EAGLE/bigram reuse, it can be up to `2 + (page_size - 1)` tokens. With
|
||||
`page_size=64`, the maximum tiny suffix is therefore `65`, matching the
|
||||
observed hang log.
|
||||
- This does not mean the user prompt changed; it is an artifact of the current
|
||||
cache key contract.
|
||||
|
||||
## C24: Tiny page-count cache-hit suffixes still enter NSA in-seq CP split
|
||||
|
||||
Finding:
|
||||
|
||||
- Latest remote log `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_140440.log`
|
||||
shows C22 is active: the repeated cache-hit request with
|
||||
`prefix_lens=[40320]`, `extend_lens=[65]`, `page_size=64`, `cp_size=8`
|
||||
skips both MLA and index async prefetchers:
|
||||
`create_skip reason=extend_below_min ... min_async_extend_tokens=512`,
|
||||
followed by `create_result ... has_mla=False has_index=False`.
|
||||
- The same request then hangs before any `forward_probe` / `forward_layer`
|
||||
line for layer 2; the next relevant lines are detokenizer heartbeat timeout.
|
||||
- The earlier log `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_134616.log`
|
||||
showed the same 65-token repeated suffix hanging when async prefetchers were
|
||||
created. Therefore async prefetch creation is not the root cause.
|
||||
- `can_cp_split()` still allows NSA in-seq CP split whenever
|
||||
`sum(extend_seq_lens_cpu) >= cp_size`. For the observed repeated request,
|
||||
page-aligned physical coverage is only `ceil(65 / 64) = 2` pages, far below
|
||||
`cp_size=8`.
|
||||
- `build_page_aligned_in_seq_split_list(total_len=128, extend_len=65,
|
||||
cp_size=8)` produces a degenerate split where only two page units are real and
|
||||
most CP lanes get zero valid suffix tokens. This is the common condition
|
||||
across both hang logs.
|
||||
|
||||
Correction:
|
||||
|
||||
- Async prefetch gating alone is insufficient. CP shared-KV page-aligned
|
||||
in-seq split itself should require at least one physical suffix page per CP
|
||||
rank: `ceil(extend_len / page_size) >= cp_size`.
|
||||
- Below that threshold, run the request without NSA in-seq CP split rather than
|
||||
generating mostly-zero CP segments. This keeps the cache page contract
|
||||
intact while avoiding the fragile zero-lane distributed path.
|
||||
- This must not disable target current reuse. With no async prefetcher, the
|
||||
backend should still use synchronous prefix materialize + current splice for
|
||||
cache-hit suffixes, and full current reuse for cache-miss/current-only
|
||||
requests.
|
||||
|
||||
Implementation:
|
||||
|
||||
- `can_cp_split()` now skips CP shared-KV in-seq split when a page-aligned
|
||||
cache-hit suffix has fewer padded suffix pages than CP lanes. For example
|
||||
`extend_len=65,page_size=64,cp_size=8` has two physical suffix pages and no
|
||||
longer creates mostly-zero CP segments.
|
||||
- `can_reuse_current_extend_kv()` now treats tail-padded `out_cache_loc` as
|
||||
eligible when it still contains all valid current suffix rows. The backend
|
||||
slices partial-current `current_kv_cache/current_locs` down to the valid
|
||||
suffix length before synchronous prefix compose, so padding rows do not become
|
||||
current page-slot data.
|
||||
- Current-only/full-current reuse remains unsliced in the existing path because
|
||||
padded current-only CP requests already rely on the padded query/cache shape.
|
||||
|
||||
Verified C24:
|
||||
|
||||
- Remote container `/sgl-workspace/sglang-tai` on `g0034`:
|
||||
- `python -m py_compile` for `utils.py`, `cp_shared_kv_runtime.py`, and
|
||||
`nsa_backend.py` passed.
|
||||
- Red tests first failed on the old code:
|
||||
`test_can_cp_split_skips_cp_when_radix_hit_suffix_has_too_few_pages`,
|
||||
`test_can_cp_split_skips_cp_when_page_units_do_not_cover_all_lanes`, and
|
||||
padded-current-reuse coverage.
|
||||
- After the fix, the targeted C24 tests passed: `4 passed`.
|
||||
- Related unit suite passed:
|
||||
`test_nsa_cp_utils.py`, `test_cp_shared_kv_runtime.py`,
|
||||
`test_cp_shared_kv_layout.py` => `128 passed, 5 warnings, 2 subtests passed`.
|
||||
|
||||
## C25: 2026-05-29 remote "hang" snapshot is idle prefill plus L1 owner-lane eviction churn
|
||||
|
||||
Finding:
|
||||
|
||||
- Latest remote log `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_143924.log`
|
||||
did not show a prefill scheduler deadlock at the time of inspection:
|
||||
- `/health` responded `200 OK`;
|
||||
- metrics reported `num_running_reqs=0`, `num_queue_reqs=0`,
|
||||
`num_prefill_prealloc_queue_reqs=0`;
|
||||
- detokenizer `py-spy` stack was idle in `recv_pyobj`;
|
||||
- scheduler ranks were in the normal idle receive path:
|
||||
CP0 sampling at `recv_requests()` nonblocking ZMQ receive and CP1-7
|
||||
sampling at `broadcast_pyobj()` from `recv_requests()`.
|
||||
- The last completed user POST in that log was at `2026-05-29 14:52:08`.
|
||||
After that, the log only showed periodic health/internal tiny one-token
|
||||
prefill batches, so the prefill process itself had no visible user request
|
||||
pending.
|
||||
- The performance collapse immediately before that idle period coincided with
|
||||
repeated L1/device owner-lane eviction and load-back churn, not host-L2
|
||||
eviction:
|
||||
- no `deterministic CP host eviction before write`,
|
||||
`_evict_host_for_physical_slots`, or host-full fallback appeared in the
|
||||
log after startup;
|
||||
- `HiCache-load` repeatedly had low per-owner availability and evicted device
|
||||
victims before CP load-back, e.g. node `873` loaded `101632` host tokens
|
||||
but first evicted victims `[627,604]` for `102016` device tokens;
|
||||
- generic device eviction then also evicted large backed nodes for much
|
||||
smaller capacity deficits, e.g. `num_tokens=10752` evicted `79488`, and
|
||||
`num_tokens=19968` evicted `91200`.
|
||||
|
||||
Implication:
|
||||
|
||||
- The current evidence points to L1/device cache thrashing under CP owner-lane
|
||||
constraints once the working set is saturated. Host cache may be nearly full
|
||||
externally, but this log's hot path is not host eviction; it is repeated
|
||||
host-hit load-back requiring device-owner-lane free pages, followed by
|
||||
coarse whole-node device eviction.
|
||||
- The existing CP load-back owner-lane planner avoids random lane mismatches,
|
||||
but its eviction granularity is still whole radix nodes. When leaves are
|
||||
large, a small lane deficit can evict a much larger cached prefix, destroying
|
||||
future L1 hits and increasing H2D load-back work.
|
||||
- The generic compute-owner allocation eviction path still calls
|
||||
`tree_cache.evict(EvictParams(... owner_lane_deficits=...))`, but
|
||||
`owner_lane_deficits` is not consumed by `HiRadixCache.evict()`. It therefore
|
||||
remains a coarse total-token eviction path and can over-evict when only a few
|
||||
owner lanes are short.
|
||||
|
||||
Next correction target:
|
||||
|
||||
- Do not treat this snapshot as an async-prefetch hang. The next fix should
|
||||
reduce saturated-L1 owner-lane churn:
|
||||
1. add an owner-lane-aware device eviction path for compute-owner allocation,
|
||||
reusing the CP load-back victim selection logic where possible;
|
||||
2. cap or skip load-back when the required owner-lane eviction cost is far
|
||||
larger than the host-hit benefit, unless the cache-hit prefix is large
|
||||
enough to amortize the churn;
|
||||
3. longer term, split/schedule radix leaves at page-aligned boundaries before
|
||||
async backup/load so eviction can free page-sized or small-node units
|
||||
instead of whole long-request leaves.
|
||||
|
||||
## C26: 2026-05-29 hang is a scheduler collective-order divergence, not only idle L1 churn
|
||||
|
||||
Finding:
|
||||
|
||||
- A later remote stack sample on `g0034` for
|
||||
`/mnt/beegfs/cjy/sglang_cp_hicache_20260529_143924.log` showed a real
|
||||
distributed scheduler hang:
|
||||
- CP0 / CP6 were in `recv_requests()->broadcast_pyobj()`;
|
||||
- CP1 / CP2 / CP5 / CP7 were in
|
||||
`scheduler_dp_attn_mixin.all_gather()` from
|
||||
`prepare_mlp_sync_batch_raw()->maybe_prepare_mlp_sync_batch()`;
|
||||
- CP3 was in idle memory check and CP4 in the prefill event-loop boundary;
|
||||
- detokenizer was idle in `recv_pyobj`.
|
||||
- This means ranks disagreed on the next collective: some entered the DP/attn
|
||||
MLP-sync `all_gather`, while others entered scheduler request broadcast.
|
||||
That is a collective-order divergence and can deadlock even when queues and
|
||||
metrics look empty.
|
||||
|
||||
Immediate debugging target:
|
||||
|
||||
- Read `event_loop_normal_disagg_prefill`,
|
||||
`get_next_disagg_prefill_batch_to_run`, and
|
||||
`scheduler_dp_attn_mixin.maybe_prepare_mlp_sync_batch` to identify how one CP
|
||||
rank can decide it has a batch requiring MLP sync while another decides it is
|
||||
idle and returns to `recv_requests`.
|
||||
- The earlier owner-lane eviction churn remains a performance issue, but it is
|
||||
not sufficient to explain this collective-order mismatch.
|
||||
|
||||
Correction C26:
|
||||
|
||||
- Added a single-DP idle fast path in `prepare_mlp_sync_batch_raw`: when
|
||||
`local_batch is None`, `dp_size == 1`, and `SGLANG_SCHEDULER_SKIP_ALL_GATHER`
|
||||
is not enabled, return `None` without building an idle batch or entering the
|
||||
MLP-sync all-gather.
|
||||
- Rationale: CP in-seq split runs inside one DP scheduling domain in the current
|
||||
deployment. All TP/CP ranks should have identical batch presence. An idle
|
||||
all-gather only exchanges zero metadata; after tiny internal health requests
|
||||
it can overlap with the next request broadcast and produce the observed
|
||||
split: some ranks in Gloo `broadcast_pyobj`, others in NCCL/CPU MLP-sync
|
||||
`all_gather`.
|
||||
- This does not remove MLP sync for real batches or for multi-DP cases where a
|
||||
rank may need an idle batch because another DP worker has work.
|
||||
|
||||
## C27: 2026-05-29 request failure evidence is not in the prefill log
|
||||
|
||||
Finding:
|
||||
|
||||
- In the stopped run using `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_152501.log`,
|
||||
the prefill log does not contain the request-failure traceback:
|
||||
- no `Traceback`, `RuntimeError`, scheduler-dead, health-failure, OOM, or
|
||||
Python exception lines after startup;
|
||||
- HTTP status counts in the prefill log are all `200`;
|
||||
- the last visible prefill log line is a successful health check at
|
||||
`2026-05-29 15:45:43`.
|
||||
- All prefill/decode Python workers were already stopped/zombie when inspected.
|
||||
Decode on `g0035`/`g0036` had no persisted stdout/stderr for this run; the
|
||||
process output was attached to a TTY rather than a log file. Therefore the
|
||||
concrete decode/router-side request failure cannot be reconstructed from the
|
||||
available files.
|
||||
- Decode workers on both decode hosts show zombie creation around
|
||||
`2026-05-29 15:27:26-15:27:39`, while the prefill log continued accepting
|
||||
successful `/v1/chat/completions` requests until `15:43:39` and health checks
|
||||
until `15:45:43`. This strongly suggests the user-visible request failure is
|
||||
downstream of decode/router availability or external process stop, not a
|
||||
prefill-side HTTP 4xx/5xx captured in the HiCache log.
|
||||
|
||||
Implication:
|
||||
|
||||
- Do not infer a CP HiCache correctness failure from this prefill log alone.
|
||||
The available evidence is an observability gap: decode/router failure output
|
||||
was not persisted.
|
||||
- For the next ETE run, decode/router stdout must be persisted with `tee` or an
|
||||
equivalent log sink; otherwise a stopped decode process leaves only zombie
|
||||
metadata and no Python traceback.
|
||||
|
||||
Next correction target:
|
||||
|
||||
- Continue code fixes that are already evidenced in prefill logs (owner-lane
|
||||
device eviction churn and excessive tiny/internal-request CP fallback logging),
|
||||
but treat this specific request failure as unrooted until decode/router logs
|
||||
are captured.
|
||||
|
||||
C25 correction update:
|
||||
|
||||
- Implemented owner-lane-aware `HiRadixCache.evict()` handling when
|
||||
`EvictParams.owner_lane_deficits` is set. The eviction path now plans device
|
||||
victims using owner-lane contribution, so a low-priority non-contributing
|
||||
leaf is no longer evicted before a higher-priority leaf that actually frees
|
||||
the deficit lane.
|
||||
- Reduced `_evict_for_compute_owner_lanes()` requested eviction from
|
||||
`deficit_pages * page_size * cp_size` to `deficit_pages * page_size` because
|
||||
the deficit vector is already counted in owner-lane pages. This removes the
|
||||
previous full-CP-stripe over-eviction for a single-lane deficit.
|
||||
- Remote verification on `g0034` container:
|
||||
- RED before fix:
|
||||
`test_owner_lane_evict_params_choose_deficit_contributing_victim` evicted
|
||||
the non-contributing `[8,9,10,11]` page instead of the deficit owner page.
|
||||
- GREEN after fix:
|
||||
targeted test passed.
|
||||
- Related suite passed:
|
||||
`test_cp_hicache_load_back_owner_lanes.py` => `9 passed`.
|
||||
|
||||
## C28: 2026-05-29 accept length collapse means EAGLE draft rejection, not a HiCache hit-rate counter
|
||||
|
||||
Finding:
|
||||
|
||||
- Decode `accept len` is reported from `spec_num_accepted_tokens / spec_num_forward_ct`.
|
||||
`update_spec_metrics()` adds `num_accepted_tokens + batch_size`, while
|
||||
`_resolve_spec_overlap_token_ids()` defines `num_accepted_tokens` as
|
||||
`sum(accept_lens) - len(batch.reqs)`. Therefore a logged accept length near
|
||||
`1.0` means EAGLE accepted zero draft tokens and only produced the mandatory
|
||||
target token per verify step.
|
||||
- The current remote decode processes are running with EAGLE, but not with
|
||||
`SGLANG_CP_DRAFT_SHARED_KV=1` in their environment. Prefill has
|
||||
`SGLANG_CP_DRAFT_SHARED_KV=1`, CP shared KV, HiCache, current reuse, and MLA
|
||||
prefetch enabled. Decode still includes draft KV buffers through its draft
|
||||
worker, so this is not alone proof of missing transfer, but it is an important
|
||||
deployment asymmetry to keep visible.
|
||||
- Current live metrics on the new run already show low/uneven speculative
|
||||
acceptance on some decode DP ranks (for example `spec_accept_length` around
|
||||
`1.05` on DP4/DP9 and higher on DP8/DP13), while several ranks are still at
|
||||
zero because they have not reported a recent decode window.
|
||||
|
||||
Implication:
|
||||
|
||||
- The accept-length symptom should be debugged as target/draft state divergence
|
||||
or sampling/draft-quality degradation on decode, not as a direct HiCache host
|
||||
capacity issue. The key invariant to verify is: after disaggregated prefill
|
||||
and any cache-hit/current-reuse/load-back path, target KV, draft KV, sequence
|
||||
lengths, and draft hidden state used by EAGLE are still aligned for the same
|
||||
request positions.
|
||||
- Next evidence should be low-frequency decode-side EAGLE acceptance diagnostics
|
||||
keyed by request/cache-hit status and whether draft KV buffers were received;
|
||||
per-layer prefill logs are not sufficient to prove why decode rejects drafts.
|
||||
|
||||
### C29. 2026-05-30 live run evidence: accept-len collapse is decode-side EAGLE rejection, not missing CP current reuse
|
||||
|
||||
Evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_160555.log`, `decode0_20260529_160608.log`, and `decode1_20260529_160611.log`:
|
||||
|
||||
- Decode-side `accept len: 1.00` recurs across both decode nodes. Metrics at 2026-05-30 00:16 CST showed examples: DP4=1.0 on g0035 and DP9=1.0 on g0036. This means speculative draft tokens are rejected; the accepted token is only the mandatory target token.
|
||||
- The low-accept events are not limited to requests with active disaggregation transfer. Log parsing showed many `accept len <= 1.001` events with `#transfer-req: 0`, so the symptom persists after transfer/preallocation, not only during the transfer window.
|
||||
- Prefill CP shared KV current reuse is active in the same run. Counts: `forward_partial_current_reuse=2032`, with `used_prefetch=True=1104` and `used_prefetch=False=928`. Small-prefix cases below the async-prefetch threshold still use synchronous current compose, e.g. `reason=missing_prefetcher prefix_lens=[320] ... forward_partial_current_reuse used_prefetch=False`.
|
||||
- Therefore `prefix_below_min` disables async prefix prefetcher creation only; it does not disable partial/full current reuse. It should not by itself explain accept length collapsing to 1.
|
||||
- Decode startup remains asymmetric: g0035/node-rank0 command lacks `--speculative-attention-mode prefill`, while g0036/node-rank1 includes it. Both decode launch environments only expose `SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1` and `SGLANG_DISAGGREGATION_QUEUE_SIZE=8` among CP/disagg flags.
|
||||
|
||||
Current hypothesis to test next: accept-len collapse comes from decode-side EAGLE draft/target state divergence or decode launch/config asymmetry, not from CP shared KV partial current reuse being disabled. Add low-frequency decode-side EAGLE diagnostics around accept-lens / draft KV availability / transferred request state before changing prefill cache logic again.
|
||||
|
||||
C29 correction note:
|
||||
|
||||
- `--speculative-attention-mode prefill` is the default in `ServerArgs`; therefore g0035 omitting the explicit flag and g0036 spelling it out is not enough to prove a runtime mismatch. Keep checking actual parsed server args if needed, but do not treat the command-line spelling difference alone as root cause.
|
||||
|
||||
### C30. 2026-05-30 router failures are downstream of prefill one-page cache accounting failure
|
||||
|
||||
Evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_160555.log` and router errors around `2026-05-29 16:19:08`:
|
||||
|
||||
- Router `Prefill server failed (CRITICAL)` errors align with the prefill scheduler exiting, not with an independent router/decode routing bug.
|
||||
- The direct prefill failure is the idle memory checker:
|
||||
`token_to_kv_pool_allocator memory leak detected! self.max_total_num_tokens=833024, available_size=16384, evictable_size=816576, protected_size=0, session_held=0`.
|
||||
- The accounting gap is exactly one page: `833024 - (16384 + 816576 + 0 + 0) = 64` tokens, matching the configured page size.
|
||||
- Immediately before the failure, logs show tiny internal/prefill batches where scheduler allocation is page-granular (`#new-token: 64`) but CP shared KV metadata has valid lengths such as `seq_lens=[1]` or `seq_lens=[6]`, `real_pages=1`.
|
||||
- Therefore the next root-cause target is a one-page lifecycle/accounting mismatch around tiny CP+EAGLE disagg prefill requests, especially the path `cache_unfinished_req(req)` before KV transfer and `release_kv_cache(req, tree_cache)` after transfer completion.
|
||||
|
||||
Implication:
|
||||
|
||||
- The router error is a symptom: once prefill rank workers die, decode/router cannot fetch prefill KV and reports failures.
|
||||
- This is separate from the low EAGLE `accept len` symptom. Low accept length remains a decode-side draft rejection/state divergence issue; the one-page accounting failure explains prefill death and router errors.
|
||||
|
||||
Current hypothesis to verify before fixing:
|
||||
|
||||
- Disaggregated prefill inserts and locks an unfinished radix node before transfer. On transfer completion, the generic `release_kv_cache()` path calls `cache_finished_req()` again, which can re-insert/split a finished-key view. For tiny valid lengths under page granularity, this can leave exactly one physical page outside `available + evictable + protected` accounting.
|
||||
|
||||
### C31. Repeated tiny EAGLE requests leak one CP HiCache page when exact matched tail is not page aligned
|
||||
|
||||
New root-cause evidence:
|
||||
|
||||
- A minimal remote repro with CP HiCache, `page_size=64`, EAGLE/bigram keys, and repeated `seq_len=6` requests shows the first request is balanced after release, but every repeated exact hit loses one 64-token page:
|
||||
- after first release: `avail=8128 ev=64 prot=0 sum=8192`;
|
||||
- after second release: `avail=8064 ev=64 prot=0 sum=8128`;
|
||||
- after third release: `avail=8000 ev=64 prot=0 sum=8064`.
|
||||
- This matches the live crash where `max_total_num_tokens - (available + evictable + protected + session_held) = 64`.
|
||||
- Cause: CP HiCache uses valid-token radix keys but page-granular allocator accounting. For repeated tiny EAGLE requests, the full key is an exact non-page-aligned hit, e.g. `len(keys)=5` for `seq_len=6`.
|
||||
- `cache_unfinished_req()` / `cache_finished_req()` call `_free_kv_indices_range(kv_indices, cache_protected_len, new_prefix_len)` for duplicate prefixes. In CP mode `_free_kv_indices_range` ceil-aligns `start` and floor-aligns `end` unless `include_partial_tail=True`. With `start=0,end=5,page_size=64`, the duplicate range floors to empty, so the newly allocated physical page is neither freed nor represented by a new radix node. The existing cached node accounts only its own page, leaving the new duplicate page leaked.
|
||||
|
||||
Correction target:
|
||||
|
||||
- When an insertion result is an exact full-key duplicate (`new_prefix_len == len(keys)`) under CP HiCache, free the duplicate request KV range with `include_partial_tail=True`. This frees the newly allocated physical tail page while preserving the existing cached node. Keep normal partial-split/new-node behavior page-floored to avoid freeing pages retained by radix nodes.
|
||||
|
||||
C31 correction:
|
||||
|
||||
- `RadixCache.cache_finished_req()` and `cache_unfinished_req()` now free duplicate KV ranges with `include_partial_tail=True` when CP HiCache insertion reports an exact full-key duplicate (`new_prefix_len == len(keys)`).
|
||||
- This is intentionally limited to exact duplicate hits. Partial/new-node insertions still use the page-floored free path so a newly inserted radix tail page is not accidentally returned to the allocator.
|
||||
- Added regression coverage for repeated tiny EAGLE exact hits: three repeated `seq_len=2` requests must keep `available + evictable + protected == allocator.size` after each release.
|
||||
|
||||
Verified C31:
|
||||
|
||||
- Local compile: `python -m py_compile python/sglang/srt/mem_cache/radix_cache.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py` passed.
|
||||
- Remote `g0034` container `/sgl-workspace/sglang-tai`:
|
||||
- targeted regression passed: `test_repeated_tiny_eagle_exact_hit_frees_duplicate_tail_page`;
|
||||
- related owner-lane/CP HiCache suite passed: `test_cp_hicache_load_back_owner_lanes.py => 10 passed`.
|
||||
|
||||
### C32. Decode accept-length collapse needs draft-transfer evidence, not more prefill cache changes
|
||||
|
||||
Finding:
|
||||
|
||||
- The latest persisted decode logs (`decode0_20260529_160608.log`,
|
||||
`decode1_20260529_160611.log`) show low EAGLE acceptance after transfer has
|
||||
already drained: parsing 970 decode-stat lines gives 855 lines with
|
||||
`#transfer-req: 0`, average `accept len=1.342`, and 134 exact `accept len=1.00`
|
||||
windows. Therefore the symptom is not limited to the prefill-to-decode
|
||||
transfer polling window.
|
||||
- Prefill did attach the CP draft HiCache mirror in the same run. All eight CP
|
||||
ranks logged `[HiCache-draft] attached CP draft KV host pool`, and the host
|
||||
budget was shared across target+draft. This rules out the simple hypothesis
|
||||
that prefill had no draft host mirror.
|
||||
- The logs do not prove whether the disaggregation transfer actually registered
|
||||
and copied draft KV/state buffers, because `SGLANG_CP_DRAFT_SHARED_KV_DEBUG`
|
||||
was not enabled; `draft_kv_buffer_count`, `draft_state_buffer_count`, and
|
||||
per-room draft transfer records were therefore absent.
|
||||
- Decode `accept len ~= 1` still means EAGLE draft rejection: the decode metric
|
||||
adds the mandatory target token, so `1.00` is zero accepted draft tokens.
|
||||
|
||||
Implication:
|
||||
|
||||
- Do not continue changing CP partial current reuse or HiCache load-back based
|
||||
solely on this accept-length symptom. The next evidence must come from
|
||||
decode-side EAGLE state and disaggregation draft-transfer metadata:
|
||||
transferred draft KV/state buffer counts, metadata hidden/topk presence, and
|
||||
low-frequency per-request accept-lens after transfer.
|
||||
- A safe next step is gated diagnostics, not an unconditional behavior change:
|
||||
log startup/transfer draft-buffer availability once per rank and optionally
|
||||
sample low-accept decode batches under an explicit debug env.
|
||||
|
||||
C32 correction:
|
||||
|
||||
- Added gated decode-side EAGLE acceptance diagnostics controlled by
|
||||
`SGLANG_EAGLE_ACCEPT_DEBUG=1` and sampled by
|
||||
`SGLANG_EAGLE_ACCEPT_DEBUG_INTERVAL` (default 128). The log is warning-level
|
||||
only when a sampled batch has zero accepted draft tokens; otherwise it is
|
||||
interval-sampled info. It records per-request accepted draft count, prompt
|
||||
length, output length, cached token count, PD metadata tensor presence/shape,
|
||||
and current `spec_info` topk/hidden shapes.
|
||||
- This is diagnostic-only. It does not change EAGLE sampling, KV transfer,
|
||||
cache load-back, or CP shared KV behavior.
|
||||
- Synced `environ.py` and `scheduler_output_processor_mixin.py` to remote
|
||||
`g0034:/mnt/beegfs/cjy/sglang-dev`; remote container py_compile passed.
|
||||
|
||||
C32 correction update:
|
||||
|
||||
- Extended `SGLANG_EAGLE_ACCEPT_DEBUG=1` to print one startup KV-manager audit per
|
||||
rank on both prefill and decode. The audit includes target/draft KV buffer
|
||||
counts and target/draft state buffer counts. This avoids enabling the much
|
||||
noisier `SGLANG_CP_DRAFT_SHARED_KV_DEBUG` just to answer whether draft KV/state
|
||||
transfer surfaces are registered.
|
||||
- Synced `prefill.py` and `decode.py` to remote `g0034:/mnt/beegfs/cjy/sglang-dev`;
|
||||
remote container py_compile passed for the updated files.
|
||||
|
||||
Reference in New Issue
Block a user