Files
sglang/python
leavelet 512fe92a83 Fix CP HiCache catch_up_all_layers fallback on chunked-prefill final chunk
Under overlap scheduling a chunked request's final-chunk write-backup prepare
read a stale `is_chunked` (>0): that per-tick counter is decremented in
process_batch_result, which the overlap loop runs AFTER the run_batch prepare
hook. So prepare floored `backup_end` to a page boundary (the intermediate-chunk
rule) and dropped the now-complete final tail page, while the final non-chunked
insert builds the radix node at full unaligned length. The exact-equality attach
predicate (prepared.logical_len == len(value)) then failed by
(num_tokens-1) mod page_size, dropped the per-layer overlap backup, and forced
the serial all-layer catch-up (~89% of large chunked requests in prod).

Decouple the floor decision from the stale counter: the scheduler marks
`req.cp_backup_is_intermediate_chunk = (req is self.chunked_req)` in
`_prepare_hicache_write_backups_before_forward` (the live chunked_req identity
is the authoritative "will be chunked further" signal, set this tick before
run_batch), and the prepare candidate builder floors on that flag instead of
`is_chunked`. Intermediate chunks still floor; only the misclassified final
chunk now backs up its full tail and attaches the overlap backup.

Tests: update the chunked prepare test to the new flag; add a regression test
that a final chunk with stale is_chunked reserves the full tail; add an offline
repro that drives the real prepare/insert/probe and sweeps unaligned tails.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 11:54:24 +00:00
..
2026-06-10 05:54:43 +08:00