Stabilize EAGLE draft cache hits under CP HiCache
The failing runs showed EAGLE accept length collapsing when draft cache-hit suffixes used the new partial-current splice path. This keeps target partial-current reuse enabled, but returns EAGLE/NextN draft cache-hit suffixes to the previous full-materialize path with an explicit fallback warning until the draft splice path has value-level ETE proof.\n\nThe same change set also tightens the page-granular CP HiCache contract for scheduler-visible hits and makes the prefill-to-decode EAGLE handoff observable without cloning hot-path metadata. Exact non-page CP hits are floored to a page boundary for new scheduling decisions, while internal unfinished-request refresh keeps its exact accounting.\n\nConstraint: CP shared KV and HiCache operate at page granularity; exposing token-precise CP tails to scheduler-visible cache hits can force non-page partial materialization.\nConstraint: EAGLE/NextN draft has only one executable layer, so draft prefetch and draft partial-current splice need a separate correctness contract from target layers.\nRejected: Keep draft partial-current splice enabled | remote logs correlate it with avg accept length around 0.068 and median 0.\nRejected: Clone decode metadata tensors on transfer | slot ownership until process_prebuilt consumes them avoids extra hot-path copies.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable draft partial-current reuse without metadata/draft-KV value checks and ETE accept-length evidence.\nTested: g0034 container py_compile for touched modules.\nTested: g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 183 passed, 5 warnings, 2 subtests passed.\nNot-tested: Fresh ETE accept-length run after this exact commit; requires user-driven traffic restart.
This commit is contained in:
laoyao0822
@@ -2510,3 +2510,411 @@ Interpretation:
|
||||
|
||||
- Split-prune removes a real overlapping-radix invariant violation, but it did not recover EAGLE accept length in this run.
|
||||
- The current accept collapse is more likely in the semantic correctness/content of EAGLE prebuilt state or draft/target KV alignment than in missing metadata lifetime or gross transfer failure. Next debugging should compare the actual prebuilt top-k token/content and draft input state against a non-HiCache or non-CP baseline, not just tensor shape/presence.
|
||||
|
||||
### C56. 2026-05-30 accept-length collapse investigation: current checked facts
|
||||
|
||||
Checked facts:
|
||||
|
||||
- Current decode launch uses non-v2 `EAGLEWorker`, not `EAGLEWorkerV2`: `SpeculativeAlgorithm.create_worker()` selects V2 only when overlap scheduling is enabled, while the inspected decode server args have overlap disabled. V2-only hypotheses around `prepare_for_v2_draft()` are therefore not applicable to the latest run.
|
||||
- Latest aggregate `Decode batch` accept length remains low but is not literally always 1: in the latest decode logs the average accept length is about `1.33`, median about `1.23-1.25`, with many windows at `<=1.05`. The per-request `EAGLE_ACCEPT_DEBUG` rows are intentionally zero-biased because zero-accept cases are logged much more often than non-zero samples, so they are useful for anomaly examples but not for true distribution estimates.
|
||||
- Decode processes are not running HiCache/NSA-prefill-CP-shared-KV. The low accept issue is therefore more likely caused by prefill-transferred target/draft KV or prebuilt EAGLE state content than by decode-side HiCache behavior.
|
||||
- Metadata lifetime is not the obvious failure: latest samples show `has_pd_hidden=True`, `pd_hidden_shape=(6144,)`, and `pd_topk_shape=(16,)`; missing hidden/top-k metadata is not the dominant failure mode.
|
||||
- `process_prebuilt()` builds EAGLE prebuilt state from the transferred first output token, top-k, and hidden state, then calls `cache_unfinished_req(req)`. `fill_ids` is still the prompt/input tokens prepared before transfer; for EAGLE this is plausibly intentional because unfinished bigram cache should not cache the last row that depends on the generated output. This is a checked contract point, not a fix target unless content evidence proves otherwise.
|
||||
|
||||
Next focused check:
|
||||
|
||||
- Verify whether partial cache-hit prefill has symmetric current-suffix handling for MLA KV and NSA index KV. MLA current-suffix reuse was intentionally enabled after partial cache hits; index KV current-suffix reuse may still be restricted to current-only batches. If index materialization/top-k reads a stale or incomplete persistent index buffer for a cache-hit suffix, EAGLE top-k/hidden can be semantically bad even though all transferred tensors have correct shapes.
|
||||
|
||||
### C57. 2026-05-30 accept-length collapse: target logs do not prove draft KV correctness
|
||||
|
||||
Checked facts:
|
||||
|
||||
- The current `SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH` samples are layer-probe based
|
||||
and the active probe evidence is from target layer 2. EAGLE/NextN draft has a
|
||||
single draft layer (`layer_id=0`), so these logs do not prove that draft
|
||||
partial-current compose is correct or even which draft path is active.
|
||||
- Latest prefill log does show many target cache-hit partial-current cases. For
|
||||
small cache-hit prefixes such as `prefix_lens=[320]` (5 pages), the async
|
||||
prefetcher is absent because the threshold is 8 pages, and target falls back
|
||||
to the synchronous partial-current compose path. This is expected under the
|
||||
current threshold policy and should not be re-read as “prefetch is missing”.
|
||||
- NSA index current reuse is still current-only: `_can_reuse_current_index_kv()`
|
||||
requires `is_current_only_extend_batch()`, so partial cache-hit requests do not
|
||||
take the index-current-reuse fast path. However the indexer stores the current
|
||||
suffix local index KV into the persistent pool before materializing the shared
|
||||
index buffer for top-k, so the absence of an index partial-current fast path is
|
||||
currently a performance risk more than a proven accept-length correctness root
|
||||
cause.
|
||||
- The remaining high-probability correctness surface is draft KV semantic/content
|
||||
alignment across prefill cache-hit paths and decode prebuilt consumption:
|
||||
decode itself is not running CP HiCache, so wrong draft KV transferred from
|
||||
prefill can suppress EAGLE accept while metadata tensor shapes still look
|
||||
healthy.
|
||||
|
||||
Do-not-repeat:
|
||||
|
||||
- Do not use target layer-2 MLA prefetch logs as evidence for EAGLE draft layer
|
||||
correctness. Draft needs its own low-frequency probe or a direct KV/content
|
||||
comparison.
|
||||
- Do not treat missing async prefetcher on a 5-page prefix as a bug; it follows
|
||||
the configured 8-page threshold. The correctness requirement is that sync
|
||||
partial-current/full-current still works when prefetch is disabled.
|
||||
|
||||
### C58. 2026-05-30 accept-length collapse: prebuilt/draft evidence gaps to close next
|
||||
|
||||
Checked facts:
|
||||
|
||||
- Disaggregated decode constructs the initial EAGLE running batch through
|
||||
`get_new_prebuilt_batch()`: it first calls `req.init_next_round_input(tree_cache)`,
|
||||
then `prepare_for_prebuilt()`, then `process_prebuilt()`.
|
||||
- Because decode has its own radix tree, repeated prompts can enter prebuilt with
|
||||
a non-zero decode-side `prefix_indices` even though KV has already been
|
||||
preallocated/transferred for the full prompt. Current logs do not show this
|
||||
`pre_len`, so we cannot yet correlate low accept length with decode-side
|
||||
prefix reuse.
|
||||
- `prepare_for_prebuilt()` builds `input_ids` from `fill_ids[pre_len:]`, but the
|
||||
transferred `req_to_token` rows are full-prompt rows. If any later prebuilt or
|
||||
EAGLE path consumes `out_cache_loc`, the correct transferred suffix rows start
|
||||
at `pre_len`, not at zero. This is a concrete invariant risk, but current
|
||||
non-overlap EAGLE mostly consumes the transferred req-to-token mapping and
|
||||
prebuilt EAGLE state after `prepare_for_decode()`, so it still needs runtime
|
||||
evidence before being treated as the accept-collapse root.
|
||||
- The latest target-side CP shared-KV logs still do not identify the draft layer
|
||||
path. We need a draft-layer (`is_draft_input`, layer 0) path log showing
|
||||
whether EAGLE draft is using current-only, partial-current sync/prefetch, or
|
||||
full materialize on cache-hit requests.
|
||||
|
||||
Next instrumentation:
|
||||
|
||||
- Under `SGLANG_EAGLE_ACCEPT_DEBUG`, add low-frequency decode prebuilt logs with
|
||||
`pre_len`, `extend_input_len`, `fill_len`, `origin_len`, `output_len`, and
|
||||
`cached_tokens`.
|
||||
- Under `SGLANG_EAGLE_ACCEPT_DEBUG`, add low-frequency draft MLA path logs at
|
||||
draft layer 0 with branch name, prefix/extend lengths, prefetcher presence,
|
||||
current rows, KV rows, and page-table shape. Do not use the target layer-2
|
||||
MLA prefetch probe as draft evidence.
|
||||
|
||||
### C59. 2026-05-30 latest EAGLE accept debug run: prebuilt prefix reuse is not active, draft prefill uses partial-current sync
|
||||
|
||||
Checked runtime:
|
||||
|
||||
- Latest prefill log: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_223451.log`.
|
||||
- Latest decode logs: `/mnt/beegfs/cjy/log/decode0_20260529_223458.log`, `/mnt/beegfs/cjy/log/decode1_20260529_223500.log`.
|
||||
- The new `[EAGLE_ACCEPT_DEBUG][prebuilt_prepare]` and `[EAGLE_ACCEPT_DEBUG][draft_mla_path]` probes are present in the latest run.
|
||||
|
||||
Findings:
|
||||
|
||||
- Decode prebuilt prefix reuse is not active in the inspected window: decode0 had `128` prebuilt debug rows and decode1 had `128`; all had `pre_len=0` even when transferred/cached tokens were large (`cached_max` around `106k-107k`). Therefore the suspected decode-side suffix offset from non-zero `pre_len` is not the current primary explanation.
|
||||
- EAGLE accept remains low: latest decode logs are around `avg=1.35-1.38`, `median=1.26-1.29`, `p90=1.81-1.90`, with many windows at `<=1.05`.
|
||||
- No server-side zero-output marker was found in the latest prefill/decode logs.
|
||||
- Prefill draft MLA path is now directly observed. In the latest prefill log there are `128` draft-layer path rows: `48 current_only` and `80 partial_current_sync`. No draft prefetch path is used, as intended for draft.
|
||||
- Partial-current draft examples include `prefix_lens=[320]`, large `extend_lens`, `can_current_reuse=True`, `has_prefetcher=False`, `branch=partial_current_sync`. This proves that the re-enabled draft current-suffix compose path is on the hot path for cache-hit EAGLE prefill.
|
||||
|
||||
Next focused question:
|
||||
|
||||
- Determine whether EAGLE draft partial-current compose uses the correct valid row/loc contract after `EagleDraftInput.prepare_for_extend()` shifts input ids left and appends the verified output token. If draft current rows are mapped to target-style `out_cache_loc` rows with an off-by-one semantic shift, the transferred draft KV can have healthy shapes but poor accept length.
|
||||
|
||||
### C60. 2026-05-30 likely accept-collapse root: CP-local EAGLE hidden was padded then re-split as if full hidden
|
||||
|
||||
Checked code path:
|
||||
|
||||
- `EAGLEWorker.forward_target_extend()` stores target hidden for draft in `logits_output.draft_hidden_states` when `SGLANG_CP_DRAFT_SHARED_KV=1`.
|
||||
- `DeepseekV3ForCausalLM.forward()` captures this side-channel before CP output collection, so the tensor is already CP-local.
|
||||
- `EAGLEWorker.forward_draft_extend()` passes that CP-local tensor into `EagleDraftInput.hidden_states` for the NextN draft model.
|
||||
- Before `DeepseekModelNextN.forward()` consumes it, `ForwardBatch.prepare_mlp_sync_batch()` pads `spec_info.hidden_states` to the global padded token count and saves the original tensor in `forward_batch.hidden_states_backup`.
|
||||
- `DeepseekModelNextN._get_cp_local_spec_hidden_states()` previously checked `spec_hidden_states.shape[0] == full_num_tokens` and then ran `cp_split_and_rebuild_data(...)`. After the padding above, a CP-local tensor could look like a full-token tensor and get CP-split a second time.
|
||||
|
||||
Why this matches the symptom:
|
||||
|
||||
- It affects cache-miss and cache-hit EAGLE prefill, not only HiCache partial-current paths, matching the low accept length across runs.
|
||||
- It preserves tensor shapes and transfer metadata, matching the earlier observation that hidden/top-k shapes were healthy while accept length remained bad.
|
||||
- It is introduced by the CP-local draft-hidden optimization, not by decode prebuilt prefix reuse; latest logs show `pre_len=0` for all sampled decode prebuilt rows.
|
||||
|
||||
Current local fix:
|
||||
|
||||
- `DeepseekModelNextN._get_cp_local_spec_hidden_states()` now prefers `forward_batch.hidden_states_backup` when CP draft shared KV is enabled and MLP sync padding changed the visible hidden shape.
|
||||
- The helper returns CP-local hidden directly, pads only a small local-padding gap, and only splits when the tensor is genuinely full-token hidden.
|
||||
- A low-frequency `[EAGLE_ACCEPT_DEBUG][cp_draft_hidden]` probe records which branch was taken.
|
||||
|
||||
Validation status:
|
||||
|
||||
- Local `py_compile` for `python/sglang/srt/models/deepseek_nextn.py` passed.
|
||||
- Needs remote sync/restart and ETE accept-length verification; no commit yet.
|
||||
|
||||
### C61. 2026-05-30 restart after CP-local draft-hidden fix: code path is active but accept length is still low
|
||||
|
||||
Checked runtime:
|
||||
|
||||
- Latest prefill log: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_122138.log`.
|
||||
- Latest decode logs: `/mnt/beegfs/cjy/log/decode0_20260530_122148.log`, `/mnt/beegfs/cjy/log/decode1_20260530_122154.log`.
|
||||
- The `[EAGLE_ACCEPT_DEBUG][cp_draft_hidden]` probe is present in prefill, proving the restarted process loaded the CP-local hidden backup fix.
|
||||
|
||||
Findings:
|
||||
|
||||
- The CP-local hidden backup path is active. Example rows show `padded_shape=(10120, 6144)` and `backup_shape=(1280, 6144)`, followed by `using CP-local hidden directly`.
|
||||
- EAGLE accept did **not** recover in this run:
|
||||
- `decode0_20260530_122148.log`: `n=776`, `avg=1.357`, `median=1.2`, `last100=1.323`, `min=1.0`, `max=3.35`.
|
||||
- `decode1_20260530_122154.log`: `n=754`, `avg=1.380`, `median=1.2`, `last100=1.418`, `min=1.0`, `max=3.05`.
|
||||
- Decode prebuilt prefix reuse remains inactive in the sampled window: `128` prebuilt rows per decode log and all sampled `pre_len=0`, even with non-zero `cached_tokens`.
|
||||
- No server-side zero-output marker or health/scheduler error was found in the latest prefill/decode logs.
|
||||
- Draft path remains hot and conservative: prefill has `40 current_only` and `96 partial_current_sync` draft-layer debug rows, with no draft prefetcher.
|
||||
- Only startup/empty-prefix fallback logs were observed (`prefix_len=0` for MLA/index prefetch). These are not evidence of the current accept collapse.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- C60 fixed a real CP-local hidden contract bug, but it is not sufficient to restore accept length. The next root-cause surface is value/content correctness across the EAGLE prefill-to-decode boundary, not tensor shape or decode `pre_len` offset.
|
||||
|
||||
Next focused checks:
|
||||
|
||||
- Compare the EAGLE draft artifacts produced by prefill and consumed by decode at the request boundary: transferred draft KV content/row ordering, `output_topk_p`, `output_topk_index`, and draft hidden alignment.
|
||||
- Keep the current shape/branch probes, but add only low-frequency value checksums if needed; avoid broad trace spam.
|
||||
|
||||
### C62. 2026-05-30 accept-length collapse: active worker and metadata registration facts
|
||||
|
||||
Checked facts:
|
||||
|
||||
- The latest server args show `enable_multi_layer_eagle=False` and `disable_overlap_schedule=True`, so the active speculative path is the regular non-v2 `EAGLEWorker`. Multi-layer EAGLE top-k packing hypotheses are not active for this run.
|
||||
- Prefill and decode manager startup logs both register `target_kv_bufs=78`, `draft_kv_bufs=1`, `total_kv_bufs=79`, `target_state_type=nsa`, and one appended draft NSA state buffer. This proves the draft KV/state buffer registration surface is present, but it does not prove values or row ordering are correct.
|
||||
- Latest decode `prebuilt_prepare` logs remain `pre_len=0`; decode-side radix prefix reuse is not currently shifting the prebuilt input/off-cache mapping.
|
||||
- The latest accept distribution is still low (`decode0 avg≈1.38`, `decode1 avg≈1.39`) while there are no health/scheduler errors or zero-output markers in the latest prefill/decode logs.
|
||||
|
||||
Conclusion:
|
||||
|
||||
- The next evidence gap is not worker selection, metadata shape, or missing buffer registration. We need value-level evidence at the EAGLE prefill-to-decode handoff: top-k token/probability, hidden checksum, and later draft KV content if metadata values match.
|
||||
|
||||
Next instrumentation:
|
||||
|
||||
- Add low-frequency `[EAGLE_ACCEPT_DEBUG][metadata_set]`, `[metadata_get]`, and `[prebuilt_state]` summaries under `SGLANG_EAGLE_ACCEPT_DEBUG`.
|
||||
- Keep summaries bounded: request id/room, metadata slot, output id, cached tokens, top-k head, and hidden checksum/head only. Do not add per-layer or per-token spam.
|
||||
|
||||
### C63. 2026-05-30 EAGLE handoff checksum instrumentation added
|
||||
|
||||
Local change:
|
||||
|
||||
- Added a bounded tensor digest helper in `sglang.srt.disaggregation.utils`.
|
||||
- `MetadataBuffers.set_buf()` now logs `[EAGLE_ACCEPT_DEBUG][metadata_set]` for the prefill-side EAGLE handoff slot.
|
||||
- Decode transfer commit now logs `[EAGLE_ACCEPT_DEBUG][metadata_get]` for the decode-side view of the same slot.
|
||||
- `process_prebuilt()` now logs `[EAGLE_ACCEPT_DEBUG][prebuilt_state]` after stacking top-k and hidden into the initial decode EAGLE state.
|
||||
|
||||
Purpose:
|
||||
|
||||
- If `metadata_set`, `metadata_get`, and `prebuilt_state` match for the same request/room, the low accept length is unlikely to be caused by metadata corruption and the next target should be draft KV content/row ordering.
|
||||
- If they diverge, the root is in metadata transfer/lifetime, not draft KV.
|
||||
|
||||
Verification:
|
||||
|
||||
- Local `py_compile` passed for `disaggregation/utils.py`, `disaggregation/decode.py`, `disaggregation/decode_schedule_batch_mixin.py`, `layers/attention/nsa_backend.py`, and `models/deepseek_nextn.py`.
|
||||
|
||||
Not yet verified:
|
||||
|
||||
- Needs remote sync/restart and one traffic run with `SGLANG_EAGLE_ACCEPT_DEBUG=1` to compare the actual handoff values.
|
||||
|
||||
### C64. 2026-05-30 decode metadata lifetime audit: production free exists, stale aliases remain
|
||||
|
||||
Checked facts:
|
||||
|
||||
- `SchedulerDisaggregationDecodeMixin.get_new_prebuilt_batch()` calls
|
||||
`new_batch.process_prebuilt(...)` and then releases each request's held decode
|
||||
metadata slot in a `finally` block via `_free_decode_metadata_index_if_held`.
|
||||
- The earlier suspicion that the decode metadata slot was never freed in
|
||||
production is false; the production release path exists and is covered by
|
||||
`test_get_new_prebuilt_batch_slices_waiting_queue_prefix`.
|
||||
- The release helper currently only frees the allocator index and sets
|
||||
`metadata_buffer_index=-1`. It does not clear `Req.output_topk_p`,
|
||||
`Req.output_topk_index`, or `Req.hidden_states_tensor`, so a running request
|
||||
can still retain stale views into a reusable metadata row after the slot has
|
||||
been returned to the allocator.
|
||||
|
||||
Risk:
|
||||
|
||||
- The known prebuilt consumer copies top-k and hidden state into GPU tensors
|
||||
before the slot is released, so this alias is not yet proven to be the
|
||||
accept-length root cause.
|
||||
- Keeping stale reusable-slot views on `Req` is still an unsafe lifecycle
|
||||
contract: later debug, retraction, or future prebuilt paths can observe a
|
||||
different request's EAGLE state after slot reuse.
|
||||
|
||||
Planned correction:
|
||||
|
||||
- Keep the no-clone transfer hot path, but make metadata release also detach the
|
||||
request from reusable-slot views by clearing the three EAGLE handoff fields
|
||||
after `process_prebuilt` has consumed them.
|
||||
- Preserve the existing release timing; do not move release back to
|
||||
`pop_transferred` and do not reintroduce transfer-time clones unless the new
|
||||
handoff checksums prove process-prebuilt copy is not sufficient.
|
||||
|
||||
### C65. 2026-05-30 latest remote run still lacks new metadata checksums
|
||||
|
||||
Checked facts:
|
||||
|
||||
- Remote files under `/mnt/beegfs/cjy/sglang-dev` match the local hashes for the
|
||||
new `metadata_set`, `metadata_get`, and `prebuilt_state` instrumentation.
|
||||
- The currently running logs (`sglang_cp_hicache_20260530_122138.log`,
|
||||
`decode0_20260530_122148.log`, `decode1_20260530_122154.log`) contain the
|
||||
older `draft_mla_path`, `cp_draft_hidden`, and `prebuilt_prepare` markers but
|
||||
no `metadata_set`, `metadata_get`, or `prebuilt_state` markers. Therefore the
|
||||
live process predates the latest handoff-checksum instrumentation.
|
||||
- Current accept remains low in that old process: decode0 average around 1.38
|
||||
and decode1 average around 1.39 accepted length.
|
||||
|
||||
Next evidence gate after restart:
|
||||
|
||||
- Compare `metadata_set -> metadata_get -> prebuilt_state` for the same request.
|
||||
If these match while accept remains low, the next root surface is draft KV
|
||||
content/row ordering across prefill-to-decode transfer, not metadata lifetime
|
||||
or shape.
|
||||
|
||||
C64 correction applied:
|
||||
|
||||
- `_free_decode_metadata_index_if_held` now clears `Req.output_topk_p`,
|
||||
`Req.output_topk_index`, and `Req.hidden_states_tensor` after returning the
|
||||
metadata slot to the allocator.
|
||||
- The transfer hot path still stores views and does not clone; the request owns
|
||||
the slot until `process_prebuilt` has synchronously consumed the tensors.
|
||||
- Unit expectations were extended so future changes cannot reintroduce stale
|
||||
reusable-slot aliases on long-lived decode requests.
|
||||
|
||||
C64 verification note:
|
||||
|
||||
- Local `py_compile` for the touched Python modules passed.
|
||||
- Local pytest collection for `test_decode_queue_compaction.py` is blocked in
|
||||
this environment by missing `orjson`; use the g0034 container for this test.
|
||||
|
||||
### C66. 2026-05-30 metadata checksum debug guard compatibility
|
||||
|
||||
Checked fact:
|
||||
|
||||
- The remote decode queue compaction test uses a minimal fake `spec_algorithm`
|
||||
with `is_none()` but without `is_eagle()`. The production decode transfer
|
||||
path already gates EAGLE metadata ownership on `not spec_algorithm.is_none()`;
|
||||
the new checksum-only debug branch used the narrower `is_eagle()` method and
|
||||
therefore broke this test double without changing production semantics.
|
||||
|
||||
Correction:
|
||||
|
||||
- The `[EAGLE_ACCEPT_DEBUG][metadata_get]` guard now follows the existing
|
||||
transfer ownership contract: emit only when speculative decoding is active
|
||||
(`not spec_algorithm.is_none()`) and the debug sampler allows it.
|
||||
- This is debug-only; it does not change metadata slot lifetime or request
|
||||
scheduling behavior.
|
||||
|
||||
Verification:
|
||||
|
||||
- Local `py_compile` passed for `decode.py`, the metadata release helper file,
|
||||
and the decode queue compaction test.
|
||||
- Remote container verification on g0034 passed:
|
||||
`PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py`
|
||||
-> `11 passed, 5 warnings`.
|
||||
|
||||
### C67. 2026-05-30 exact non-page CP cache hits still expose sub-page tails
|
||||
|
||||
Finding:
|
||||
|
||||
- C16/C54 floor CP radix hits when a request extends beyond an existing
|
||||
non-page-aligned valid-tail node, but an exact key hit still exposes the full
|
||||
token-precise tail.
|
||||
- In normal prefill scheduling this is still an extend: the radix key is capped at
|
||||
`input_len - 1`, so an exact EAGLE/bigram key hit with length `N` means the
|
||||
model will compute at least the next current token while the CP shared-KV
|
||||
backend sees `extend_prefix_len=N`.
|
||||
- If `N` is not page-aligned, CP shared-KV current reuse either cannot use the
|
||||
page-aligned prefix contract or must enter a mostly-zero/tiny split shape.
|
||||
This matches the recurring symptom pattern where repeated cache-hit requests
|
||||
are page-tail sized and EAGLE accept length collapses without transfer errors.
|
||||
|
||||
Decision:
|
||||
|
||||
- Treat CP HiCache radix cache as page-granular for scheduler-visible hits: an
|
||||
exact non-page valid-tail hit is floored to the previous page boundary, just
|
||||
like the extension case.
|
||||
- The sub-page tail is sacrificed and recomputed by the new request. This costs
|
||||
at most one page of reusable prefix, but keeps `prefix_indices`,
|
||||
`cache_protected_len`, prepared backup start, and CP shared-KV materialization
|
||||
on the same page-aligned contract.
|
||||
|
||||
Regression first:
|
||||
|
||||
- Added failing tests on g0034 before production changes:
|
||||
- `test_cp_match_prefix_floors_exact_valid_tail_for_exact_key`
|
||||
- `test_cp_insert_replaces_exact_valid_tail_from_page_boundary`
|
||||
- Both failed under the old helper: match exposed `[0..5]` and insert reported
|
||||
`prefix_len=6` for a 6-token node on 4-token pages.
|
||||
|
||||
C67 implementation update:
|
||||
|
||||
- Added `MatchPrefixParams.cp_floor_exact` so scheduler-visible CP prefix matches can
|
||||
floor exact non-page valid-tail hits, while the internal `cache_unfinished_req()`
|
||||
post-insert refresh can keep its own just-inserted exact tail and avoid
|
||||
invalidating current-request accounting.
|
||||
- `HiRadixCache._cp_floor_exact_valid_tail_extension_len()` now floors exact
|
||||
non-page hits when `cp_floor_exact=True` or when the request key extends beyond
|
||||
the child. Prepared-backup insertion enables the same exact-key floor so the
|
||||
recomputed page-tail backup attaches to the replacement suffix node.
|
||||
- `RadixCache.cache_unfinished_req()` calls its internal refresh match with
|
||||
`cp_floor_exact=False`; this preserves the previous exact-duplicate accounting
|
||||
fix for repeated tiny EAGLE requests.
|
||||
- Existing CP host-hit expectations were updated from token-granular `6` to
|
||||
page-granular `4` for a 6-token backed node on 4-token pages.
|
||||
|
||||
C67 verification:
|
||||
|
||||
```text
|
||||
remote g0034 container:
|
||||
PYTHONPATH=python python -m pytest -q \
|
||||
test/registered/unit/mem_cache/test_cp_hicache_metadata.py \
|
||||
test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py
|
||||
-> 109 passed, 5 warnings
|
||||
```
|
||||
|
||||
### C68. 2026-05-30 accept-length collapse correlates with draft partial-current reuse
|
||||
|
||||
Finding:
|
||||
|
||||
- Latest available remote process is no longer running; `/mnt/beegfs/cjy/log` still contains the previous run as the newest evidence:
|
||||
- Prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_122138.log`
|
||||
- Decode: `/mnt/beegfs/cjy/log/decode0_20260530_122148.log`, `/mnt/beegfs/cjy/log/decode1_20260530_122154.log`
|
||||
- Compared with the earlier baseline-ish run (`20260529_1605xx`), the failing run differs on the EAGLE/NextN draft MLA path:
|
||||
- Earlier prefill `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_160555.log`: `draft_partial_current_reuse_disabled=64`, `branch=partial_current_sync=0`.
|
||||
- Failing prefill `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_122138.log`: `draft_partial_current_reuse_disabled=0`, `branch=partial_current_sync=112`, `branch=current_only=48`.
|
||||
- Decode accept distribution regressed at the same time:
|
||||
- `decode0_20260529_160608.log`: `n=502`, `avg=1.324`, `median=1.18`, `last100=1.424`, `zero=0`.
|
||||
- `decode1_20260529_160611.log`: `n=468`, `avg=1.348`, `median=1.20`, `last100=1.351`, `zero=0`.
|
||||
- `decode0_20260530_122148.log`: `n=167806`, `avg=0.068`, `median=0.0`, `last100=0.01`, `zero=158223`.
|
||||
- `decode1_20260530_122154.log`: `n=160622`, `avg=0.068`, `median=0.0`, `last100=0.011`, `zero=151281`.
|
||||
- The failing logs predate the newest `metadata_set/get/prebuilt_state` checksum probes, so metadata handoff values are still unverified in a fresh process.
|
||||
|
||||
Working hypothesis:
|
||||
|
||||
- Target-model partial current reuse can remain enabled, but EAGLE/NextN draft partial current reuse is not yet a validated correctness contract.
|
||||
- For draft cache-hit suffixes, the safer contract is the previous full-materialize path: direct write the local draft suffix into the draft pool, then materialize the full prefix+suffix from the pool/all-rank compose. This is slower than partial splice but matched the earlier non-collapsed accept distribution.
|
||||
|
||||
Planned correction:
|
||||
|
||||
- Keep draft current-only reuse enabled for empty-prefix batches.
|
||||
- Disable only draft partial-current splice (`prefix_len > 0`) and emit an explicit `[CP_SHARED_KV_FALLBACK][current_reuse] reason=draft_partial_current_reuse_disabled` warning when this gate fires.
|
||||
- Do not add new collectives; this restores the existing full-materialize path rather than inventing a new synchronization path.
|
||||
|
||||
C68 implementation update:
|
||||
|
||||
- `should_reuse_current_extend_kv()` now returns `True` immediately for current-only batches, preserving draft and target empty-prefix reuse.
|
||||
- For partial-current batches, EAGLE/NextN draft inputs (`spec_info.is_draft_input()`) now return `False` and emit the explicit fallback warning:
|
||||
`[CP_SHARED_KV_FALLBACK][current_reuse] reason=draft_partial_current_reuse_disabled`.
|
||||
- Target partial-current reuse remains enabled.
|
||||
|
||||
C68 verification:
|
||||
|
||||
```text
|
||||
remote g0034 container:
|
||||
PYTHONPATH=python python -m pytest -q \
|
||||
test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_should_reuse_current_extend_kv_disables_draft_partial_cache_hit_suffix
|
||||
-> 1 passed
|
||||
|
||||
PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py
|
||||
-> 73 passed, 5 warnings, 2 subtests passed
|
||||
```
|
||||
|
||||
Runtime validation still needed:
|
||||
|
||||
- Start a fresh prefill/decode process with the synced code and `SGLANG_EAGLE_ACCEPT_DEBUG=1`.
|
||||
- Expected prefill evidence: draft cache-hit suffix rows should log `draft_partial_current_reuse_disabled`; draft `branch=partial_current_sync` should disappear except for old logs.
|
||||
- Expected decode evidence: accept distribution should move away from the `avg≈0.068 / median=0` collapse if C68 is the active root cause.
|
||||
|
||||
Reference in New Issue
Block a user