Cache completed chunked prefill pages in CP HiCache
Chunked prefill was skipping prepared CP HiCache backup entirely while still inserting unfinished valid-tail radix state. That combination caused middle chunks to miss reusable host cache and repeatedly hit stale-tail split fallback paths.\n\nThis keeps CP HiCache page-granular: middle chunks now reserve/write only completed pages, and unfinished radix inserts expose only page-complete prefixes while preserving the request-owned sub-page tail in prefix_indices for later chunks.\n\nConstraint: CP HiCache radix and host residency are managed at page granularity, but chunked prefill can end on a sub-page tail.\nRejected: Continue skipping chunked backup | loses middle-chunk reuse and triggers fallback storms.\nRejected: Insert sub-page chunk tails into CP HiCache | creates stale-tail prune/split conflicts against request-owned KV.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce chunked_req backup skip; keep CP HiCache chunked inserts page-complete unless the radix/host cache contract becomes sub-page aware.\nTested: python -m py_compile python/sglang/srt/mem_cache/radix_cache.py python/sglang/srt/mem_cache/hiradix_cache.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py\nTested: g0034 targeted chunked CP HiCache regression tests, 2 passed\nTested: g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py, 107 passed\nNot-tested: Fresh chunked-prefill ETE traffic run after process restart
This commit is contained in:
@@ -5088,3 +5088,65 @@ Regression test:
|
||||
attached catch-up ack `[53]`. `writing_check()` must not raise `KeyError`, must
|
||||
preserve both ack entries, and must not release the attached node behind the
|
||||
unattached head entry.
|
||||
|
||||
### C120 — 2026-06-02 Chunked prefill should cache completed middle-chunk pages
|
||||
|
||||
Finding:
|
||||
|
||||
- With chunked prefill enabled, the CP HiCache path previously skipped
|
||||
pre-forward prepared backup for every chunked request:
|
||||
`[CP_HICACHE_FALLBACK][prepare_write_backup_skipped] reason=chunked_req`.
|
||||
- `cache_unfinished_req(chunked=True)` still inserted scheduler-visible valid
|
||||
tails. For CP HiCache this is the wrong granularity: the cache is managed at
|
||||
page granularity, so inserting a non-page-aligned chunk tail creates a stale
|
||||
tail that a later chunk must prune while the same request can still own or
|
||||
lock those KV slots.
|
||||
- Remote fallback storms showed this exact shape: many
|
||||
`prepare_write_backup_skipped` and `insert_deferred_pending_backup_split`
|
||||
events, often with `ongoing=0 pending=0`, meaning the split was blocked by
|
||||
node protection/refs rather than an actual active transfer.
|
||||
|
||||
Requirement:
|
||||
|
||||
- Intermediate chunks are not disposable. They should write completed pages
|
||||
into CP HiCache to maximize reuse and avoid doing all host backup only at the
|
||||
final chunk.
|
||||
- The current incomplete page must remain request-owned until a later chunk or
|
||||
final insert makes it page-complete. CP HiCache should not create a radix
|
||||
node for a sub-page valid tail.
|
||||
|
||||
Correction:
|
||||
|
||||
- `prepare_write_backup_for_req()` no longer skips chunked requests. For
|
||||
chunked CP HiCache requests it floors the backup end to the nearest completed
|
||||
page and reserves/writes only `[start, floor_to_page_len(len(keys)))`.
|
||||
- `cache_unfinished_req(chunked=True)` now inserts only completed pages when CP
|
||||
HiCache is active. If no new completed page exists beyond
|
||||
`cache_protected_len`, it keeps `req.prefix_indices` pointing at the full
|
||||
request-owned KV span and returns without mutating the radix tree.
|
||||
- The sub-page tail is deliberately retained in `req.prefix_indices` but not in
|
||||
the CP HiCache radix tree. This trades at most one page of temporary
|
||||
non-cacheable tail for stable page-granular radix/host-cache state.
|
||||
|
||||
Regression tests:
|
||||
|
||||
- `test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_prepare_write_backup_for_req_chunked_reserves_completed_pages`
|
||||
verifies a 10-token chunk with `page_size=4` reserves only the first 8
|
||||
completed tokens instead of warning and skipping backup.
|
||||
- `test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_cp_cache_unfinished_chunked_inserts_only_completed_pages`
|
||||
verifies `cache_unfinished_req(chunked=True)` inserts only 8 page-completed
|
||||
tokens while preserving full 10-token `req.prefix_indices` for the scheduler.
|
||||
|
||||
Validation:
|
||||
|
||||
- Remote targeted regression tests on `g0034`:
|
||||
`2 passed, 3 warnings`.
|
||||
- Remote full metadata suite:
|
||||
`PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py`
|
||||
passed with `107 passed, 5 warnings`.
|
||||
|
||||
Remaining validation gap:
|
||||
|
||||
- Requires a fresh ETE run with chunked prefill enabled to verify that the
|
||||
fallback storm disappears in production traffic. An already-running remote
|
||||
process will not pick up this change until restarted.
|
||||
|
||||
Reference in New Issue
Block a user