Commit Graph
30 Commits
Author SHA1 Message Date
ThomasX 099bcfb41e fix: copy spec_info in ScheduleBatch.copy() to preserve Eagle hidden states in overlap prefill
In disaggregated prefill with --enable-single-batch-overlap,
batch.copy() was used to defer process_batch_result, but the copy
omitted spec_info. This caused req.hidden_states_tensor to be None
when using Eagle/spec-v2 speculative decoding on the decode server,
silently degrading draft quality for the first decode round.
2026-05-11 04:44:25 +08:00
ThomasX 3f1e5c5896 fix: initialize cp_shared_kv_layout for draft model runners
The CpSharedKVLayout initialization was gated behind
`if not self.is_draft_worker`, leaving the draft model runner
with `uses_cp_shared_kv=True` but `cp_shared_kv_layout=None`.
This caused an AssertionError in NSAIndexer._filter_shared_index_write
during DRAFT_EXTEND under CP shared KV + MTP.

Move the layout init after the `is_draft_worker` guard so both
target and draft model runners have it available.
2026-05-11 02:51:22 +08:00
ThomasX 29a395fe95 feat: add structured debug logging for CP HiCache write/evict/load paths
Add logger.info statements with [CacheCtrl-write], [MemCache-evict], [MemCache-alloc], [HiCache-write], [HiCache-evict], [HiCache-load] tags across cache_controller.py, common.py, and hiradix_cache.py to improve observability of cache operations.
2026-05-11 00:00:04 +08:00
ThomasX 3a5928ef53 feat: add page-aligned index validation for CP HiCache direct transfers
Add validate_page_aligned_token_indices utility and apply it across
HiCache write/load paths, NSA indexer transfers, and CUDA direct copy
kernels to reject malformed (partial, misaligned, non-contiguous) page
groups before they reach native transfer code. Also validate supported
CP HiCache backend/layout combinations at server startup.
2026-05-10 02:02:54 +08:00
ThomasX f38937723e fix: address CP HiCache review regressions 2026-05-08 04:11:25 +08:00
ThomasX 8844c303e2 fix: synchronize CP HiCache host eviction semantics 2026-05-08 03:20:47 +08:00
ThomasX eecc8e21ec docs: defer CP HiCache performance follow-up 2026-05-08 03:07:18 +08:00
ThomasX 9617d52639 fix: keep HiCache storage config runtime import 2026-05-08 03:03:10 +08:00
ThomasX 3e9152194b test: cover CP HiCache host integration 2026-05-08 02:59:27 +08:00
ThomasX 39f40c02b4 fix: handle empty HiCache match prefixes 2026-05-08 02:47:25 +08:00
ThomasX 829f3ebceb feat: restore CP HiCache host hits as logical locs 2026-05-08 02:34:16 +08:00
ThomasX f10a8983c3 fix: keep CP HiCache load host indices on CPU 2026-05-08 02:26:46 +08:00
ThomasX 94df7a6909 feat: map CP HiCache loads to owned physical slots 2026-05-08 02:16:04 +08:00
ThomasX 08d518c2ed fix: preserve CP host siblings during stale cleanup 2026-05-08 02:06:04 +08:00
ThomasX 2df832dec0 fix: unblock CP host eviction past stale leaves 2026-05-08 01:59:14 +08:00
ThomasX f1a7e605e8 feat: split and evict CP HiCache metadata 2026-05-08 01:50:07 +08:00
ThomasX bebbbfd0e8 fix: treat CP HiCache metadata as backup state 2026-05-08 01:41:53 +08:00
ThomasX 08a95a35a6 fix: implement CP HiCache write retry eviction 2026-05-08 01:35:55 +08:00
ThomasX 068aa2f910 feat: track CP HiCache write-back state 2026-05-08 01:29:39 +08:00
ThomasX a8816f18dd test: clean CP HiCache controller imports 2026-05-08 01:23:12 +08:00
ThomasX 301420f976 feat: map CP HiCache writes to owned physical slots 2026-05-08 01:14:58 +08:00
ThomasX eb0ec4299d test: cover CP HiCache metadata validation edges 2026-05-08 01:09:32 +08:00
ThomasX 1ede8bb999 test: harden CP HiCache metadata invariants 2026-05-08 01:06:58 +08:00
ThomasX 5fbe43381f feat: add CP HiCache node metadata 2026-05-08 01:00:09 +08:00
ThomasX da4994ec26 fix: fail fast for CP HiCache storage args 2026-05-08 00:55:35 +08:00
ThomasX 54f8cab7e7 test: enforce early CP HiCache storage rejection 2026-05-08 00:50:52 +08:00
ThomasX ffcbf904e4 fix: reject CP shared KV with HiCache storage 2026-05-08 00:42:08 +08:00
ThomasX 36ee803aa7 test: fix Task 1 CPU unit coverage 2026-05-08 00:29:33 +08:00
ThomasX 95bacb8862 fix: preserve HiCache load-back allocator headroom 2026-05-08 00:18:33 +08:00
ThomasX 1f074f434e docs: add CP HiCache host plan 2026-05-08 00:12:39 +08:00