Keep tiny CP cache-hit suffixes off async prefetch

The repeated-request hang reproduced on g0034 after the cached-prefix
request created both MLA and index async prefetchers with
prefix_lens=[40320] and extend_lens=[65]. The existing one-page default
only blocked sub-page suffixes, so a barely-over-one-page suffix still
entered the next-layer collective path before any later forward progress
was logged.

Raise the default async prefetch extend gate to one page per CP lane while
keeping the env override. This only gates async prefetcher object
creation; target partial-current reuse still uses the synchronous
page-slot compose/current-splice path when no prefetcher exists.

Constraint: cp_size=8,page_size=64 repeated prompt had extend_len=65 and hung immediately after has_mla=True has_index=True create_result logs
Rejected: Disable partial-current reuse for short extends | that would lose the cache-hit benefit and regress current/full reuse
Rejected: Disable all async prefetch by default | broader performance impact than the observed tiny-suffix failure
Confidence: medium
Scope-risk: moderate
Directive: Do not lower the default below one page per CP lane without ETE proof that repeated cache-hit tiny suffixes no longer hang
Tested: Remote g0034 container py_compile for touched runtime/prefetch/test files; targeted C22 tests passed 2 tests plus 2 subtests; full test_cp_shared_kv_runtime.py passed 73 tests plus 2 subtests
Not-tested: Full multi-node ETE repeated-request run after this threshold change
This commit is contained in:
laoyao0822
2026-05-29 22:04:01 +08:00
parent 40cf691c78
commit 21065cdfdf
4 changed files with 135 additions and 59 deletions

View File

@@ -1490,8 +1490,10 @@ Correction:
- If partial-current reuse is intentionally rejected, for example the current
conservative draft/EAGLE cache-hit guard, that rejection must not disable
current-only/full-current reuse for other request shapes.
- Default the async gate to one page when `page_size` is known; allow an env
override to lower or raise the threshold for experiments.
- C21 initially defaulted the async gate to one page when `page_size` is known;
C22 supersedes that default with one page per CP lane after the 65-token
repeated-request hang showed that one page was still too small. The env
override remains available for experiments.
- This must not force tiny cache-hit suffixes back to compact full
materialization.
@@ -1509,3 +1511,50 @@ Verified:
- Remote container `/sgl-workspace/sglang-tai` on `g0034`:
- `test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py`: `73 passed`
- `test/registered/unit/mem_cache/test_cp_hicache_metadata.py`: `90 passed`
## C22: One-page async prefetch gate still admits repeated-request hangs
Finding:
- Latest remote log `/mnt/beegfs/cjy/sglang_cp_hicache_20260529_134616.log`
shows the first 40k request completed and backed up CP HiCache node 4.
- The repeated request hit the cached prefix and created both MLA and index
prefetchers on every CP rank:
`prefix_lens=[40320]`, `extend_lens=[65]`, `prefix_pages=630`,
`min_async_extend_tokens=64`, `has_mla=True has_index=True`.
- No `index_start_request` or `forward_probe` appears after those `create_result`
lines; the next lines are detokenizer heartbeat timeout and shutdown cleanup.
- Therefore C21's one-page default threshold is insufficient. It blocks
sub-page suffixes but still allows a 65-token suffix, i.e. just over one page,
to enter async next-layer prefetch.
Correction:
- The intended production default is CP-lane scale, not one page: with
`cp_size=8,page_size=64`, async MLA/index prefetch should require at least
`8 * 64 = 512` current extend tokens unless explicitly overridden.
- This matches the owner-lane mental model: below one page per CP rank, the
prefetch collective and page-slot compose overhead is not amortized and is a
correctness risk while padded-tail ETE coverage is still incomplete.
- This gate only controls async prefetcher object creation. Target
partial-current reuse and full current reuse must continue through the
synchronous page-slot compose/current-splice path when no prefetcher exists.
Required test update:
- Change the min-extend helper contract from "one page" to "one page per CP
lane" when both `cp_size` and `page_size` are known.
- Add a regression case for `extend_len=65,page_size=64,cp_size=8` proving both
MLA and index prefetchers return `None` before touching pool getters/remap
builders.
Verified C22:
- Remote container `/sgl-workspace/sglang-tai` on `g0034`:
- `python -m py_compile` for `cp_shared_kv_runtime.py`,
`cp_shared_kv_prefetch.py`, and `test_cp_shared_kv_runtime.py` passed.
- Targeted C22 tests passed: `2 passed, 2 subtests passed`.
- Full `test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py` passed:
`73 passed, 2 subtests passed`.
- Local pytest remains blocked by missing optional deps such as `orjson`; remote
container is the authoritative verification environment for this path.