Prepared per-layer CP HiCache backups can enqueue their final write ack before the radix node is attached. Chunked prefill exposed this window when a separate catch-up backup made writing_check drain the ack queue and pop an unattached node id.
Keep ready but unattached prepared acks queued until insert either attaches the prepared backup or rollback removes the orphan ack. Also document the reactive host free-room eviction plan separately from this state-machine fix.
Constraint: CP HiCache prepared backup transfer can complete before radix insertion attaches node state
Rejected: Drop unknown ack ids | would orphan a later successful prepared attach and leak write state
Rejected: Chunked-only guard | the invalid assumption is in the generic CP write ack state machine
Confidence: high
Scope-risk: narrow
Directive: Do not drain CP write acks unless every ack id is registered in pending_host_backups or ongoing_write_through
Tested: Remote red-green test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_writing_check_defers_unattached_prepared_ack
Tested: Remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py (116 passed)
Tested: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py
Not-tested: Full chunked-prefill ETE replay after this commit
Co-authored-by: OmX <omx@oh-my-codex.dev>
Chunked prefill can revisit a sub-page CP HiCache tail while a per-layer backup is still in flight. The old insert path split the radix node first and only then tried to prune the stale tail, so an unprunable pending backup raised after tree mutation and propagated to the scheduler.
This makes split/prune atomic from the radix-tree perspective: drain completed write acks only on the conflict path, preflight pending/unprunable state before split, and return a deferred insert result when the backup is still in flight. Unfinished requests keep their KV ownership for transfer/release instead of rematching or freeing pages under a stale tree state.
Constraint: CP HiCache backup metadata is node/page owned and cannot be repartitioned while per-layer D2H is pending
Rejected: Split the pending backup node | would require repartitioning in-flight backup metadata and host reservations
Rejected: Delete the stale tail unconditionally | risks freeing device/host pages still owned by pending backup
Confidence: high
Scope-risk: moderate
Directive: Do not move stale-tail prune after split without a preflight; pending backup split conflicts must remain non-mutating
Tested: remote g0034 container py_compile for mem_cache files
Tested: remote g0034 PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py => 115 passed
Tested: local git diff --check
Not-tested: full chunked prefill ETE after service restart
Co-authored-by: OmX <omx@oh-my-codex.dev>
The LPF direct-kernel work exposed an allocation-dependent performance boundary: layer-page-first reduces descriptors only when host pages have contiguous extents. Capture the benchmark evidence and owner-lane allocator interpretation so future changes do not rediscover the same constraint.
Constraint: Current CP compute-owner allocation selects pages by modulo owner lane, e.g. (page_id - 1) % cp_size.
Rejected: Document LPF as an unconditional production improvement | random and owner-lane same-layout benchmarks are neutral without compact allocation.
Rejected: Treat owner_lane benchmark as full allocator replay | it models the modulo-lane constraint, while free/release history can add fragmentation.
Confidence: medium
Scope-risk: narrow
Directive: Tie any production LPF switch to compact/extent-aware host allocation or gather/staging support.
Tested: Local doc marker check for C115/C116/C117.
Tested: Remote tai-kernel CUDA pytest for the referenced kernel/test changes -> 45 passed in 2.52s.
Not-tested: Full SGLang ETE after changing host allocation policy; no such policy change is included.
NSA FP8 CP shared-KV reuse must operate on packed page-slot rows, not bf16 compact rows. The change keeps current-only and partial-current reuse inside the page-aligned materialization contract, fails fast for non-page-aligned CP split inputs, and prevents FP8 FlashMLA-KV prefill from reaching incompatible in-seq CP metadata.
Constraint: NSA FP8 persistent MLA KV rows are packed 656-byte records and CP shared KV cache management is page-granular.\nConstraint: FlashMLA-KV prefill metadata is not CP-local after NSA in-seq splitting.\nRejected: Silently splice bf16 current rows into FP8 materialized cache | corrupts the packed cache layout.\nRejected: Keep FP8 FlashMLA-KV auto prefill under NSA CP | reaches num_splits shape errors after q-row splitting.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable FP8 FlashMLA-KV prefill for NSA in-seq CP until metadata is rebuilt after CP splitting or made CP-local.\nTested: Local git diff --check and py_compile for touched SGLang files.\nTested: Remote g0034 related unit sweep recorded in docs: test_nsa_cp_utils.py, test_cp_shared_kv_layout.py, test_cp_shared_kv_runtime.py, test_cp_hicache_metadata.py passed.\nNot-tested: Full FP8 ETE startup and performance run after this commit.
Launching CP shared-KV prefetch from MLA prepare or the indexer can make
next-layer prefix work overlap current-layer MQA/materialization instead of
the attention window. Centralize the launch in the NSA backend after
current-layer materialization and before attention, and leave the early
indexer hook inert so the call site cannot regress silently.
The accompanying notes capture the draft-as-forward-layer follow-up plan and
the latest OOM diagnosis: the observed 178945-token failure matches the CP
in-seq MQA logits allocation, so the follow-up fix is q-dimension chunking in
_get_topk_ragged_with_cp(), not a max-prefetch-size gate.
Constraint: CP shared-KV prefetch must avoid overlapping current-layer MQA/materialization and must not add silent fallback behavior.
Rejected: Limit maximum prefetch size | hides the CP logits peak and can reduce cache/prefetch effectiveness.
Rejected: Keep prepare/indexer launch sites | they place next-layer collectives in the wrong overlap window.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce early CP prefetch launch without checking Nsight overlap and CP MQA memory peaks.
Tested: Local git diff --check and py_compile for touched Python files.
Tested: Remote container py_compile plus targeted pytest: 3 passed, 5 warnings.
Not-tested: Full ETE under production traffic after this commit.
Not-tested: CP in-seq MQA logits chunking; documented as follow-up.
Synchronous CP shared KV full-hit and partial-current paths can now use the tai-kernel CUDA IPC slot-dense materialize path instead of first copying local owner pages and then running a dense CP all-reduce. This keeps the async prefetch pipeline unchanged while routing the safer synchronous runtime path through descriptor-driven owner-page reads.
The runtime builds descriptors from the existing page-aligned slot contract, caches peer pointer tables for long-lived KV/index buffers, and falls back with explicit warnings if the tai-kernel IPC capability is missing. Unit coverage locks offset handle exchange, descriptor construction, and both MLA/index full and partial-current calls.
Constraint: Async prefetch currently has unresolved scheduling contention with GEMM/MoE and current-layer KV collectives, so this commit intentionally does not wire IPC into prefetch.
Constraint: tai-kernel must provide offset-aware CUDA IPC symbols from af9fb67.
Rejected: Replace prefetch all-reduce in this slice | Nsight shows prefetch timing and communicator contention need a separate scheduling design.
Rejected: Fail-fast on missing tai IPC immediately | remote ETE still needs to validate production deployment capability before removing the warning fallback.
Confidence: medium
Scope-risk: moderate
Directive: Keep prefetch and synchronous materialize decisions separate until prefetch has a low-SM/copy-engine schedule.
Related: tai-kernel af9fb67
Tested: git diff --check; remote runtime/unit evidence recorded in docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md.
Not-tested: Fresh GLM5 ETE after this commit; async prefetch IPC path.
Co-authored-by: OmX <omx@oh-my-codex.dev>
The materialize transport direction changed after measurement: direct CUDA IPC peer-page materialize was initially slow because of an under-parallel launch policy, not because slot-dense materialize was inherently too expensive. Capture the corrected evidence and the remaining production constraints before SGLang runtime wiring starts.
Constraint: Current near-term goal accepts SM-consuming kernels for low latency/high throughput; low-SM prefetch friendliness is deferred.
Rejected: Treat the first slow IPC result as a design blocker | tuned launch parameters beat dense all-reduce across the measured 4k-120k prefix range.
Rejected: Switch consumers to owner-concat immediately | slot-dense fused materialize is competitive enough to preserve the current consumer contract for the next integration step.
Confidence: medium
Scope-risk: narrow
Directive: Keep this document updated with every benchmark/result correction to avoid re-litigating stale conclusions.
Tested: Documentation update based on remote g0034 tai-kernel CUDA tests and cp_shared_kv_ipc_sm_tuned_20260531_181926 benchmark log
Not-tested: SGLang runtime ETE serving with IPC materialize enabled
Add a production-style CP shared KV collective benchmark and keep the page-aligned cache contract ledger current with the collective, unpack, multi-extend prefix, P2P, and CUDA IPC findings. The benchmark makes the final consumer layout explicit so dense all-reduce and owner-packed all-gather are compared after producing the same logical-dense result.\n\nConstraint: Prefixes may be assembled from multiple radix/cache extents, so single-run zigzag ordering is only a gated fast path.\nRejected: Treat rank-major all-gather output as the final product | consumers require logical-dense ordering.\nRejected: Switch index materialization based on noisy microbenchmarks | current evidence is not strong enough for production.\nConfidence: medium\nScope-risk: narrow\nDirective: Do not replace generic logical-dense materialization with zigzag ordered gather unless a run descriptor proves the prefix shape.\nTested: python -m py_compile benchmark/hicache/bench_cp_shared_kv_production_collective.py\nTested: Remote production-style benchmark logs under /mnt/beegfs/cjy/log/cp_shared_kv_collective_bench_*_20260531_*.log\nNot-tested: Production SGLang ETE with a CUDA IPC/P2P materialization path; this commit only adds benchmark/documentation surfaces.
Page-aligned CP shared KV can pad out_cache_loc beyond valid current rows, so current reuse now gates MLA composition on the valid extend rows and permits draft partial-current reuse once the TAI sparse-page capability check passes. The TAI current-slot path self-tests sparse pages before use and falls back to the torch reference when the installed kernel is stale.
Eviction success and no-op diagnostics were also moved from INFO to DEBUG so owner-lane and host-admission churn does not flood production logs; true write failures remain WARNING.
Constraint: CP shared KV uses page-aligned physical reservations where valid suffix rows can be shorter than padded out_cache_loc.
Constraint: Production failure/fallback logs must remain visible, but hot successful eviction paths should not emit INFO per victim/rank.
Rejected: Keep draft partial-current reuse disabled | would preserve avoidable full materialization on draft cache-hit suffixes.
Rejected: Trust the TAI current-slot kernel unconditionally | stale kernels can corrupt sparse current-page composition.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce INFO logging in eviction hot paths without rate limiting and runtime evidence.
Tested: local py_compile for touched Python files
Tested: local git diff --check
Tested: remote container py_compile for touched Python files
Tested: remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py::TestHiCacheEvictLoggingLevels::test_evict_hot_path_success_logs_are_debug_only test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 82 passed, 5 warnings, 2 subtests passed
Not-tested: full ETE traffic after this commit; draft partial-current accept length still needs user-driven runtime validation
The index path now mirrors the target MLA partial-current contract: prefetched or synchronously materialized prefix pages are composed with valid current index K/scale rows in slot-dense page buffers. Current-only batches keep the compact current-index fast path, while partial cache-hit batches share one composed dense index buffer across the in-seq prev/next topk pair.\n\nThe prefetch consume path remaps through the slot page inverse instead of treating the slot-dense buffer as physical-pool capacity, and current-index quantization uses valid extend rows so padded out_cache_loc does not disable reuse.\n\nConstraint: CP shared KV remains page-slot based; padding rows must stay invisible to attention/index semantics\nConstraint: Draft/EAGLE partial-current reuse remains guarded by should_reuse_current_extend_kv\nRejected: Replace prefix all-reduce with all-gather | NCCL all-gather still uses SM and would require an additional compose/scatter step\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce current-only gating for index reuse; partial target cache hits must compose prefix + valid current rows\nTested: Local py_compile for touched Python files\nTested: g0034 sglang-glm5-dev-2 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 77 passed, 5 warnings, 2 subtests passed\nNot-tested: Full ETE traffic with latest commit; CUDA perf impact of index partial-current prefetch under production load
CP HiCache direct/page_first_direct all-layer backup was still able to enter sgl-kernel's stale cudaMemcpyBatchAsync path, which segfaults under CUDA 13 before Python can surface an error. The SGLang route now avoids that all-layer sgl-kernel op for page_first_direct backup and uses the TAI per-layer direct LF->PF op for MHA, MLA, and NSA indexer data.\n\nThe load-back path also prepares NSA indexer page indices once per load op and reuses them across per-layer H2D loads, preserving per-layer overlap while removing redundant page-index derivation.\n\nConstraint: Remote runtime is CUDA 13.0 where sgl-kernel's all-layer direct LF->PF op uses the wrong cudaMemcpyBatchAsync ABI.\nRejected: Patch sgl-kernel in this branch | we are converging production HiCache direct/page_first_direct paths onto tai-kernel and do not want to maintain another CUDA-ABI-sensitive copy path here.\nRejected: Collapse H2D load-back into one all-layer op | that would reduce submit count but lose per-layer completion visibility and forward overlap.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce sgl_kernel.transfer_kv_all_layer_direct_lf_pf for direct/page_first_direct HiCache backup without CUDA 13 ABI verification.\nTested: g0034 container: PYTHONPATH=python python -m pytest -q -s test/registered/unit/mem_cache/test_nsa_pool_host_unit.py -> 10 passed, 3 warnings.\nTested: g0034 container: PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py -> 61 passed, 3 warnings.\nTested: python -m py_compile python/sglang/srt/mem_cache/memory_pool_host.py python/sglang/srt/managers/cache_controller.py test/registered/unit/mem_cache/test_nsa_pool_host_unit.py test/registered/unit/managers/test_hicache_controller_cp.py\nNot-tested: Full ETE prefill/decode traffic after this commit.\nNot-tested: sgl-kernel implementation itself remains unchanged.
The failing runs showed EAGLE accept length collapsing when draft cache-hit suffixes used the new partial-current splice path. This keeps target partial-current reuse enabled, but returns EAGLE/NextN draft cache-hit suffixes to the previous full-materialize path with an explicit fallback warning until the draft splice path has value-level ETE proof.\n\nThe same change set also tightens the page-granular CP HiCache contract for scheduler-visible hits and makes the prefill-to-decode EAGLE handoff observable without cloning hot-path metadata. Exact non-page CP hits are floored to a page boundary for new scheduling decisions, while internal unfinished-request refresh keeps its exact accounting.\n\nConstraint: CP shared KV and HiCache operate at page granularity; exposing token-precise CP tails to scheduler-visible cache hits can force non-page partial materialization.\nConstraint: EAGLE/NextN draft has only one executable layer, so draft prefetch and draft partial-current splice need a separate correctness contract from target layers.\nRejected: Keep draft partial-current splice enabled | remote logs correlate it with avg accept length around 0.068 and median 0.\nRejected: Clone decode metadata tensors on transfer | slot ownership until process_prebuilt consumes them avoids extra hot-path copies.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable draft partial-current reuse without metadata/draft-KV value checks and ETE accept-length evidence.\nTested: g0034 container py_compile for touched modules.\nTested: g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 183 passed, 5 warnings, 2 subtests passed.\nNot-tested: Fresh ETE accept-length run after this exact commit; requires user-driven traffic restart.
CP HiCache owns KV at page granularity, but exact valid-tail extension and backed partial-tail split could leave an old sub-page tail child beside a new suffix that reuses the same physical page. That makes radix residency ambiguous across device, host, and draft mirrors. The insert/match split paths now prune stale floored tails when safe, and defer/fail through the existing pending-split path when the subtree is protected or has in-flight backup state.\n\nThis also keeps a temporary scheduler boundary warning for externally observed zero-output responses so future ETE runs can classify whether zero visible output reaches SGLang's output processor.\n\nConstraint: CP shared KV and HiCache manage physical KV by page, while radix keys retain valid-token lengths.\nRejected: Keep overlapping old tail nodes after page-floor split | leaves two independent cache states for one physical tail page.\nRejected: Force-prune protected or in-flight backup tails | can mutate cache state still used by active transfer or inference.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not remove the stale-tail prune without replacing it with another page-granular ownership rule for CP HiCache radix splits.\nTested: Remote py_compile for hiradix_cache.py and scheduler_output_processor_mixin.py in g0034 container.\nTested: Remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py -> 97 passed, 5 warnings.\nNot-tested: Full ETE recovery of EAGLE accept length; latest ETE still shows accept collapse, documented in C55.\nNot-tested: Router/client-side output_len=0 correlation when server-side OUTPUT_ZERO_DEBUG does not fire.
Transferred EAGLE metadata buffers are reusable slot views. The previous clone-based mitigation protected correctness but added copies on the transfer hot path and hid the actual lifetime contract. This change makes the transferred request own the metadata slot while it sits in the decode waiting queue, then releases it immediately after process_prebuilt has consumed top-k and hidden state into the prebuilt batch. Abort paths also release any held decode metadata slot.
Constraint: Decode disaggregation metadata buffers are reusable slot views consumed later by process_prebuilt.
Rejected: Clone transferred EAGLE tensors at commit time | correct but less efficient and masks the ownership contract.
Rejected: Release in process_batch_result_prebuilt | holds slots across forward longer than needed.
Confidence: medium
Scope-risk: moderate
Directive: Do not free successful EAGLE transfer metadata in pop_transferred unless process_prebuilt consumption is also moved earlier.
Tested: Remote py_compile for decode.py, scheduler.py, scheduler_output_processor_mixin.py, and test_decode_queue_compaction.py.
Tested: Remote focused lifecycle tests passed: 3 passed.
Tested: Remote full test_decode_queue_compaction.py passed: 11 passed, 5 warnings.
Not-tested: Fresh ETE runtime validation of EAGLE accept-length recovery after the C48 sync.
Decode committed EAGLE top-k and hidden-state tensors as views into reusable metadata-buffer rows. The metadata index is freed immediately after transfer commit, while the request may wait before process_prebuilt consumes the draft state. Under concurrent cache-hit traffic a later transfer can overwrite the same row, leaving output_id copied correctly but EAGLE draft state corrupted, which matches low accept length despite successful KV/state registration.
Constraint: Metadata slots are intentionally recycled right after transfer commit for throughput.
Rejected: Hold metadata slots until process_prebuilt | larger lifetime change and reduces transfer capacity; cloning the small prebuilt EAGLE state is narrower.
Confidence: high
Scope-risk: narrow
Directive: Do not store reusable metadata-buffer views on Req unless the slot lifetime is extended through all consumers.
Tested: Local py_compile for decode.py and test_decode_queue_compaction.py.
Tested: Remote g0034 container py_compile for decode.py and test_decode_queue_compaction.py.
Tested: Remote g0034 focused clone-lifetime test: 1 passed.
Tested: Remote g0034 test_decode_queue_compaction.py: 10 passed, 5 warnings.
Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
Cache-hit EAGLE/NextN draft was falling back to full materialization while the target path used page-aligned prefix materialize plus fresh current-suffix splice. That creates a target/draft KV source asymmetry exactly on the high-cache-hit path where decode accept length collapsed. The draft model still does not get next-layer async prefetch; only the same-layer current suffix compose contract is made role-agnostic.
Constraint: EAGLE/NextN has no real next decoder layer, so draft async prefetch remains disabled.
Rejected: Restore draft full-materialize fallback | recreates the observed target/draft cache-hit asymmetry and hides stale-current-suffix bugs.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce draft cache-hit current-reuse fallback without proving the draft persistent pool has fully fresh suffix rows before attention.
Tested: Local py_compile for cp_shared_kv_runtime.py and test_cp_shared_kv_runtime.py.
Tested: Remote g0034 container py_compile for changed runtime/test files.
Tested: Remote g0034 pytest test_cp_shared_kv_runtime.py: 73 passed, 5 warnings, 2 subtests passed.
Tested: Remote g0034 related suite test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py: 128 passed, 5 warnings, 2 subtests passed.
Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
The C32 startup audit was diagnostic-only, but it incorrectly read spec_algorithm from helper queue objects. PrefillBootstrapQueue and DecodePreallocQueue keep scheduler-owned runtime state instead, so the debug path failed before KV manager initialization could complete.\n\nUse scheduler.spec_algorithm for both startup audits and record the ownership rule in the page-aligned cache ledger.\n\nConstraint: Queue helpers do not copy every scheduler field onto self.\nRejected: Disable EAGLE accept diagnostics | would lose the draft-transfer evidence needed for the accept-len investigation.\nConfidence: high\nScope-risk: narrow\nDirective: Disaggregation helper diagnostics should read scheduler-owned runtime state unless the constructor explicitly copied the field.\nTested: Local py_compile for prefill.py and decode.py.\nTested: Local git diff --check.\nTested: Synced prefill.py/decode.py to g0034 and remote container py_compile passed.\nNot-tested: Full disaggregated startup and ETE traffic after this fix; user controls runtime launch and load generation.
CP shared KV and HiCache now keep page-aligned physical ownership while preserving valid-token radix semantics. Repeated tiny EAGLE exact hits free duplicate tail pages instead of leaking one allocator page, owner-lane load-back uses page-vector admission/eviction, and single-DP idle schedulers avoid entering an unnecessary MLP-sync collective.
The commit also records the current page-aligned cache contract and adds gated decode-side EAGLE accept diagnostics so future accept-length collapses can be tied to draft KV/state transfer evidence instead of more prefill cache speculation.
Constraint: CP HiCache allocator ownership is page-granular while radix matching remains valid-token based.
Constraint: New diagnostics must be gated and must not alter normal EAGLE, transfer, or cache behavior.
Rejected: Padding short requests to cp_size or 2*cp_size pages | wastes KV capacity and still hides valid-tail lifecycle bugs.
Rejected: Adding more unconditional collectives to prove CP consistency | hot-path collectives previously caused severe performance risk.
Confidence: medium
Scope-risk: broad
Directive: Do not reintroduce silent fallback for CP shared KV/HiCache paths; warning-level fallback or fail-fast is intentional.
Tested: git diff --check
Tested: local py_compile for all modified Python files
Tested: remote g0034 container py_compile for modified Python/test files
Tested: remote g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/managers/test_scheduler_dp_attn_mixin.py => 114 passed, 5 warnings, 2 subtests passed
Not-tested: full ETE traffic rerun after this commit
Not-tested: CUDA/TAI kernel benchmark coverage for all production shapes
The repeated-request hang reproduced on g0034 after the cached-prefix
request created both MLA and index async prefetchers with
prefix_lens=[40320] and extend_lens=[65]. The existing one-page default
only blocked sub-page suffixes, so a barely-over-one-page suffix still
entered the next-layer collective path before any later forward progress
was logged.
Raise the default async prefetch extend gate to one page per CP lane while
keeping the env override. This only gates async prefetcher object
creation; target partial-current reuse still uses the synchronous
page-slot compose/current-splice path when no prefetcher exists.
Constraint: cp_size=8,page_size=64 repeated prompt had extend_len=65 and hung immediately after has_mla=True has_index=True create_result logs
Rejected: Disable partial-current reuse for short extends | that would lose the cache-hit benefit and regress current/full reuse
Rejected: Disable all async prefetch by default | broader performance impact than the observed tiny-suffix failure
Confidence: medium
Scope-risk: moderate
Directive: Do not lower the default below one page per CP lane without ETE proof that repeated cache-hit tiny suffixes no longer hang
Tested: Remote g0034 container py_compile for touched runtime/prefetch/test files; targeted C22 tests passed 2 tests plus 2 subtests; full test_cp_shared_kv_runtime.py passed 73 tests plus 2 subtests
Not-tested: Full multi-node ETE repeated-request run after this threshold change
CP shared KV HiCache now treats non-page-aligned valid-tail nodes as page-owned when a later request extends beyond them. The prefix probe, match path, insert path, and prepared backup start now agree on flooring the reusable prefix to the previous physical page boundary, so prepared backup metadata cannot start mid-page or fail to attach at insertion.
Duplicate frees under CP HiCache now go through a page-safe free helper. Insert and unfinished duplicate ranges free only fully unprotected pages; no-insert completion still releases the right tail owned only by the finishing request.
Constraint: CP HiCache allocator frees whole physical pages even when called with token-granular locs.
Rejected: Partial-page sharing/refcounting | too complex for the current page-as-minimum-unit contract.
Rejected: Fix only prepare_write_backup_for_req | match_prefix and insert would still expose exact valid-tail hits and desynchronize prepared backup length.
Confidence: medium
Scope-risk: moderate
Directive: Do not expose non-page-aligned CP valid-tail hits to extending requests unless partial-page ownership is explicitly implemented end-to-end.
Tested: remote g0034 py_compile for touched files
Tested: remote g0034 test_cp_hicache_metadata.py 97 passed
Tested: remote g0034 test_cp_shared_kv_runtime.py 73 passed
Not-tested: test_cp_shared_kv_layout.py aborts during installed sgl_kernel architecture-specific op loading before assertions
Async MLA/index prefetch is a scheduling optimization, not the correctness contract for target current reuse. Tiny cache-hit suffixes can skip async prefetcher creation while target partial-current reuse still composes page-slot prefix materialization with current KV rows synchronously. CP HiCache radix/device accounting now treats retained valid-tail pages as physical page spans so allocator state stays consistent when logical cache keys are shorter than the retained page.
Constraint: CP shared KV ownership and HiCache residency are page-granular while request-visible cache lengths remain valid-token lengths.
Constraint: Async prefetch can hang or regress on large-prefix tiny-extend traffic and must not be required for current reuse.
Rejected: Treat missing prefetcher as fail-fast for target partial-current reuse | disabled useful current reuse and broke tiny-prefix/tiny-suffix traffic.
Rejected: Keep async prefetcher object with synchronous consume mode | conflates prefetch object existence with current-layer correctness and hides fallback semantics.
Confidence: medium
Scope-risk: moderate
Directive: Do not make current-only or target partial-current reuse depend on MLA/index prefetcher creation; prefetcher objects mean async next-layer work exists.
Tested: Remote g0034 container py_compile for touched modules.
Tested: Remote g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 73 passed.
Tested: Remote g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py -> 90 passed.
Not-tested: Latest full ETE traffic run with GLM-5.1 CP HiCache after this commit.
Not-tested: CUDA kernel-level performance impact of synchronous no-prefetch partial-current compose.
Backed CP HiCache tail splits were floored during match, but write preparation and radix insertion could still observe the raw sub-page prefix. That could skip one token during pre-forward reservation or call CpHiCacheNodeMetadata.split() with a non-page boundary. The shared helper now floors backed partial-tail prefixes before probe, match, and insert.
Constraint: CP HiCache uses page-granular physical ownership while radix keys may keep valid-tail lengths.
Rejected: Let probe and insert keep token-precise tail prefixes | it reintroduces half-page ownership through a different entry point.
Confidence: high
Scope-risk: moderate
Directive: Any CP radix path that can split or reserve against backed HiCache metadata must use the same page-floor policy.
Tested: Remote red tests first for probe and insert failures in g0034 container.
Tested: Remote py_compile for hiradix_cache.py and test_cp_hicache_metadata.py.
Tested: Remote targeted C7 tests: 4 passed, 3 warnings.
Tested: Remote pytest test_cp_hicache_metadata.py test_cp_hicache_load_back_owner_lanes.py: 94 passed, 5 warnings.
Not-tested: Live ETE traffic with concurrent divergent tail prompts.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP HiCache ownership is page-granular, so a backed radix node must not be split inside a padded physical tail page. When a shorter hit would require an interior tail-page boundary, matching now floors to the previous page boundary and sacrifices the sub-page cache prefix instead of splitting ownership metadata.
Constraint: Host/device/draft CP HiCache metadata tracks page owners and padded physical spans.
Rejected: Split one padded tail page across two radix nodes | it would require half-page ownership semantics and risks double-counting capacity.
Confidence: high
Scope-risk: moderate
Directive: Keep exact valid-tail hits, but floor partial backed-node splits to page boundaries unless metadata gains explicit sub-page ownership.
Tested: Remote py_compile for hiradix_cache.py and test_cp_hicache_metadata.py in g0034 container.
Tested: Remote pytest targeted backed-tail split tests plus exact valid-tail hit test: 3 passed.
Tested: Remote pytest test_cp_hicache_metadata.py test_cp_hicache_load_back_owner_lanes.py: 92 passed, 5 warnings.
Not-tested: Live ETE traffic under divergent short-prefix prompts.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Mixed prefix/current MLA reuse depends on the page-slot prefetch compose path to preserve padded tail semantics. The compact materialize/current merge path can re-expose suffix slack as valid dense rows, so the backend now raises with an explicit fail-fast marker instead of silently falling back.
Constraint: Page-aligned CP shared KV contract requires suffix tail slack to remain invalid.
Rejected: Keep compact merge as a fallback | it has a different dense-row contract and can hide correctness bugs.
Confidence: high
Scope-risk: moderate
Directive: Do not reintroduce merge_materialized_and_current_kv on MLA partial-current reuse without a page-slot correctness proof.
Tested: Remote py_compile for nsa_backend.py and test_nsa_cp_utils.py in g0034 container.
Tested: Remote pytest test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_mla_partial_current_path_fails_fast_instead_of_compact_fallback.
Tested: Remote pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py: 124 passed, 5 warnings.
Not-tested: Live ETE traffic and CUDA kernel execution for this fail-fast path.
Co-authored-by: OmX <omx@oh-my-codex.dev>
C11-C14 are now tied to explicit remote verification instead of leaving the ledger at the intermediate failure state. The mixed-version symbol issue is documented as a caller-sync problem, and the stale C14 expectations are recorded as intentional current contracts rather than open regressions.
Constraint: Local pytest in this workspace is blocked by missing optional runtime dependencies such as orjson, so remote g0034 container tests are the authoritative CPU/unit evidence for these files.
Rejected: Reintroduce cp_shared_kv_mla_prefetch_should_trace_tiny_extend | the current callers no longer import it, and adding a dead compatibility helper would hide sync mistakes.
Confidence: high
Scope-risk: narrow
Directive: Keep C13 validation focused on matching caller/runtime sync; do not resurrect stale helper APIs unless an active caller contract requires them.
Tested: Local py_compile for touched runtime/prefetch/caller/test files; remote caller import check printed CALLER_IMPORT_OK; remote exact C14 set passed 6 passed, 3 warnings; remote three-file suite passed 123 passed, 5 warnings.
Not-tested: CUDA/TAI ETE runtime logs were not rerun in this documentation-only commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Current reuse, TAI materialize, and TAI fused MLA store are optimized CP shared-KV paths. Their fallback helpers already emitted warnings, but the messages were not consistently grep-able with the standard CP shared-KV fallback marker.
This moves the marker into the helper layer so individual fallback call sites do not have to remember to include it, and keeps the existing per-reason rate limiting.
Constraint: Runtime fallback logs must be visible without enabling debug logging.
Rejected: Prefix only selected call sites | helper-level prefixing avoids future unmarked fallback messages.
Confidence: high
Scope-risk: narrow
Directive: New CP shared-KV runtime fallback helpers should use [CP_SHARED_KV_FALLBACK] and a component name.
Tested: Local py_compile for cp_shared_kv_runtime.py and test_cp_shared_kv_runtime.py.
Tested: Local extracted assertLogs check for all three fallback helper prefixes.
Tested: Remote g0034 pytest exact runtime helper tests: 2 passed, 3 warnings.
Not-tested: Full cp_shared_kv_runtime.py suite in this slice.
Co-authored-by: OmX <omx@oh-my-codex.dev>
The compute-owner allocation path is an intended hot path for CP shared KV. Falling back to legacy page allocation changes the behavior profile enough that INFO/no-marker logging is too easy to miss during profiling.
This makes the fallback warning-visible and gives it the standard CP shared-KV fallback marker while preserving the existing behavior of logging every fallback event.
Constraint: Production fallbacks from intended CP shared-KV hot paths must be visible and grep-able.
Rejected: Keep INFO logging | historical silent or low-visibility fallbacks hid inactive optimized paths.
Confidence: medium
Scope-risk: narrow
Directive: Keep fallback logs warning-level unless the path is proven to be expected steady state and separately observable.
Tested: Local py_compile for common.py and test_cp_shared_kv_layout.py.
Tested: Local extracted assertLogs check for _log_cp_shared_kv_alloc_fallback warning prefix and reason formatting.
Tested: Remote g0034 py_compile for common.py and test_cp_shared_kv_layout.py.
Not-tested: Full remote pytest for test_cp_shared_kv_layout.py; current g0034 container aborts while importing installed sgl_kernel common_ops before test execution.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Mooncake is the only disaggregation transfer backend in this branch with CP shared-KV owner filtering plus logical-page-position destination selection. NIXL still slices destination pages by the original chunk slice, so allowing CP shared-KV prefill on NIXL can silently pair filtered prefill pages with the wrong decode pages.
This keeps the supported path narrow while preserving the page-aligned transfer contract: non-page-aligned valid tails transfer their physical tail page, but do not get padded to CP-size pages.
Constraint: CP shared-KV transfer remaps prefill logical pages to per-rank physical pages while decode metadata remains request-position based.
Rejected: Let NIXL continue through the generic slice path | it lacks logical-page-position selection and can silently corrupt CP shared-KV transfers.
Confidence: high
Scope-risk: narrow
Directive: Do not enable CP shared-KV on another PD transfer backend until its sender filters owner pages and selects decode pages by logical request-page position.
Tested: Local py_compile for server_args and touched tests.
Tested: Remote g0034 pytest test_cp_shared_kv_transfer_mapping.py test_req_to_token_pool.py TestHiCacheArgs: 22 passed, 8 subtests passed.
Not-tested: End-to-end PD transfer with a live non-page-aligned prompt.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Draft HiCache already piggybacks target CP reservations, but the controller coverage only exercised page-aligned nodes. The new tests pin the intended contract for valid-tail nodes: target and draft reserve/load the same padded physical page while scheduler-visible load results keep only valid logical locs.
Constraint: Draft KV is a target mirror; it must not choose an independent valid-only host path.
Rejected: Re-enable draft partial-current reuse in this slice | current safe contract keeps EAGLE/NextN cache-hit draft on full materialization until same-layer padded visibility is proven.
Confidence: medium
Scope-risk: narrow
Directive: Keep draft and target host metadata coupled; do not add draft-only capacity or prefetch decisions.
Tested: local py_compile for test_hicache_controller_cp.py.
Tested: remote g0034 test_hicache_controller_cp.py: 59 passed, 3 warnings.
Not-tested: CUDA E2E runtime for this commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Load-back owner-lane eviction can need to evict a device-resident node whose radix value is a valid tail shorter than the physical tail page. The eviction planner now pads that value only to the page boundary before deriving owner counts, matching the existing write/load capacity contract without exposing padding to radix or scheduler lengths.
Constraint: CP HiCache capacity remains page-owner based while radix node values remain valid-token based.
Constraint: Avoid collectives; owner-lane capacity must be deterministic from local metadata and logical page ids.
Rejected: Require device-resident victim values to be page-aligned | valid-tail cache nodes are now an intentional supported state.
Rejected: Pad to cp_size pages | this would waste KV and violate the page-boundary-only contract.
Confidence: medium
Scope-risk: narrow
Directive: If split-inside-tail support is added later, preserve page ownership/refcount semantics before sharing one padded physical page across radix nodes.
Tested: local py_compile for hiradix_cache.py and touched CP HiCache tests.
Tested: remote g0034 new C8 exact tests: 3 passed, 3 warnings.
Tested: remote g0034 CP HiCache impacted suites: 146 passed, 5 warnings.
Tested: remote g0034 CP shared KV C1-C5 suite: 122 passed, 5 warnings.
Not-tested: full local pytest, blocked by missing runtime dependencies such as starlette.
Not-tested: CUDA E2E runtime for this commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP HiCache now keeps radix and scheduler-visible lengths as valid tokens while host/device transfers reserve and replay the padded physical page span. Exact valid-tail write, insertion, and match paths no longer fall back to page-flooring; the physical owner-lane contract still uses padded page metadata.
Constraint: Scheduler prefix indices must never include padded tail locs.
Constraint: Host/device transfer and owner-lane admission remain page-based.
Rejected: Pad to cp_size or 2*cp_size pages | wastes KV and recreates short-tail fallback behavior.
Rejected: Expose padded locs through load_cp return | would leak fake tokens into req.prefix_indices.
Confidence: medium
Scope-risk: moderate
Directive: Do not implement split-inside-tail by duplicating page_owners without a page-sharing/refcount design.
Tested: local py_compile for touched CP HiCache/radix/controller files and tests.
Tested: remote g0034 CP HiCache impacted suites: 143 passed, 5 warnings.
Tested: remote g0034 CP shared KV C1-C5 suite: 122 passed, 5 warnings.
Not-tested: full local pytest, blocked by missing runtime dependencies such as orjson/starlette.
Not-tested: CUDA E2E runtime for this commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared KV and HiCache need a stable contract where physical cache coverage is page-aligned, while scheduler/radix-visible hit length remains the valid token length. This records the contract, adds page-aligned extent metadata, keeps owner assignment on actual tail pages instead of short-prefix fallback, and updates partial current reuse tests around tail-page masking.
Constraint: CP owner lanes operate on page units while scheduler and radix hit accounting must remain token-valid.
Rejected: Pad short suffixes to cp_size or 2*cp_size pages | wastes KV capacity and can turn a small tail into a much larger physical span.
Rejected: Silent direct-write or prefetch fallback | production fallback must be warning-visible for diagnosis.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce replicated short-radix fallback without checking docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md.
Tested: local py_compile for touched runtime, utility, owner, and unit-test files.
Tested: remote g0034 container three-file suite: 122 passed, 5 warnings.
Not-tested: full local pytest, blocked by missing runtime dependencies such as orjson.
Not-tested: CUDA E2E runtime for this commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared KV and HiCache now use owner-lane metadata as the
authoritative capacity view for host write admission and GPU load-back
planning. This removes the debug scalar capacity env and keeps CP load-back
from relying on a rank-wide scalar collective when per-owner availability is
already known. The load-back planner also accounts for evicting child leaves
that unlock ancestor device residency, which fixes small lane deficits despite
large aggregate evictable capacity.
The commit also adds gated CPU timing logs for CP shared-KV MLA/index
prefetch and a CUDA microbenchmark for comparing dense all-reduce with
owner-packed all-gather layouts. The timing logs are intentionally behind the
existing MLA prefetch log env and should not be enabled for throughput
measurements.
Constraint: CP shared KV owner lanes require target/draft capacity decisions to preserve page_owners rather than total-token scalars
Constraint: CUDA collective benchmarks must run on target GPU hosts, not locally
Rejected: Keep SGLANG_CP_HICACHE_CAPACITY_DEBUG observer env | owner-lane admission now replaces that scalar debug path
Rejected: Add a silent scalar-allreduce fallback | unexpected owner-lane mismatch should fail fast or log loudly
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce CP capacity collectives on the scheduler hot path without proving the owner-lane metadata is insufficient
Directive: Disable SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH for end-to-end performance runs; it is diagnostic and high-volume
Tested: git diff --check
Tested: python -m py_compile on changed runtime/test/benchmark Python files
Tested: remote pytest -q test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py (81 passed, 5 warnings)
Not-tested: CUDA benchmark benchmark/hicache/bench_cp_shared_kv_prefetch_collective.py
Not-tested: full GLM5 E2E throughput after this commit
CP shared KV cannot treat capacity as a scalar token count: cache-hit load-back and fresh extend allocation both have to preserve the logical page owner pattern or later direct writes, HiCache reload, and prefix materialization can read the wrong lane. This change moves the critical paths to owner-lane plans, makes owner-lane exhaustion recoverable during prefill scheduling, and routes shared-KV prefix prefetch through prefetch-stream-safe KV getters so HiCache layer-load waits do not attach to the forward stream.
Constraint: CP shared KV correctness depends on page owner lane preservation across allocation, backup, load, eviction, and prefix materialization.
Constraint: Avoid adding CP/global collectives for capacity agreement; derive capacity from deterministic local owner-lane state.
Rejected: Keep SGLANG_DISABLE_TAI_OWNER_SELECT fallback | legacy allocation can silently break owner-lane invariants.
Rejected: Scalar total-token eviction for CP HiCache load-back | total capacity can be sufficient while the required owner lane is exhausted.
Confidence: medium
Scope-risk: broad
Directive: Do not reintroduce silent legacy fallback in owner-lane paths; unexpected owner-lane failure must be warning-level fail-closed or recoverable capacity wait.
Tested: Remote g0034 container PYTHONPATH=python python -m pytest test/registered/unit/mem_cache/test_alloc_pages_with_owners.py test/registered/unit/mem_cache/test_cp_shared_kv_layout.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -q -> 95 passed.
Tested: Local py_compile for modified runtime/cache/scheduler modules.
Not-tested: Full CUDA ETE performance trace for cache-hit overlap and MTP accept-rate impact.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP HiCache write-through under shared KV was still using rank-wide collectives to decide host reservation eviction, and per-layer backup registration could be bypassed before the final forward boundary. This moves backup registration to the final run_batch pre-forward boundary, forwards it through SessionAwareCache, exposes fallback paths as explicit warnings, and introduces deterministic owner-lane capacity planning for CP host reservation.
Constraint: CP shared-KV ranks must keep target and draft host reservations owner-lane consistent without adding hot-path collective synchronization
Constraint: Remote CUDA validation must run in the g0034 container, not locally
Rejected: Keep reserve_slots_max all_reduce as the default admission path | observed reserve collectives reaching double-digit and occasional 100ms+ latency
Rejected: Silent post-forward catch-up backup | hides when per-layer forward-overlap backup is not actually active
Confidence: medium
Scope-risk: broad
Directive: Do not reintroduce CP HiCache hot-path collectives without a measured mismatch case and explicit fallback warning
Tested: py_compile for modified Python modules and CP HiCache metadata test file in remote g0034 container
Tested: python3 -m pytest test/registered/unit/mem_cache/test_cp_hicache_metadata.py -q in remote g0034 container (75 passed, 5 warnings)
Tested: git diff --check HEAD~1..HEAD
Not-tested: Local pytest blocked by missing pybase64 in the local environment
Not-tested: Full CP HiCache + MTP E2E after the no-collective reservation change
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared KV with HiCache and EAGLE needs host backup to overlap forward while keeping radix visibility synchronous. The change reserves host slots before forward, drives target and draft backup from explicit layer-end hooks, and commits host visibility only after the final target/draft ack. It also probes the final insertion prefix before early reservation so repeated EAGLE prompts do not prepare duplicate suffix backups that later rollback as insert_miss.
Constraint: CP ranks use independent shared-KV pools, so target/draft host state must remain atomically visible at the radix boundary.
Constraint: Fused MLA and NSA store paths can bypass store-side notifier hooks, so layer end is the safer backup progress boundary.
Rejected: Store-side backup notifier as the primary trigger | fused store and zero-local paths made notifier coverage fragile.
Rejected: Reserve from cache_protected_len alone | EAGLE bigram/page alignment can make final insertion find a longer existing prefix and force duplicate rollback work.
Confidence: medium
Scope-risk: moderate
Directive: Do not add per-layer CP collectives here; keep radix state synchronous and data transfer asynchronous/local-event driven.
Tested: local git diff --check
Tested: local py_compile for touched CP HiCache/cache-controller/deepseek/test files
Tested: remote pytest test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/managers/test_hicache_controller_cp.py -q (115 passed, 5 warnings)
Not-tested: full GLM5 ETE server rerun after this commit
CP shared KV needs HiCache backup to overlap with layer execution without exposing partially copied host state. Split CP backup into reservation, pending radix state, per-layer target/draft D2H submission, and one final ack-driven visibility commit. The all-layer path remains available only as an explicit fallback and now logs a warning when used.
Constraint: CP shared KV owner-lane metadata and draft/MTP KV must stay strongly synchronized with target KV.
Constraint: Local CUDA tests are disallowed; CUDA verification was run only in the g0034 container.
Rejected: Let target layer hooks copy draft KV too | draft may not have stored that layer yet, which can corrupt MTP accept behavior.
Rejected: Silent all-layer fallback | it hides performance regressions and makes ETE logs ambiguous.
Confidence: medium
Scope-risk: broad
Directive: Reserved or partially copied host payloads must remain invisible until final ack commits pending_host_backups.
Tested: g0034 docker /sgl-workspace/sglang-tai PYTHONPATH=/mnt/beegfs/cjy/tai-kernel/python:python python -m pytest test/registered/unit/managers/test_hicache_controller_cp.py -q -> 49 passed.
Tested: g0034 docker /sgl-workspace/sglang-tai PYTHONPATH=/mnt/beegfs/cjy/tai-kernel/python:python python -m pytest test/registered/unit/mem_cache/test_cp_hicache_metadata.py -q -> 58 passed.
Tested: g0034 docker /mnt/beegfs/cjy/tai-kernel PYTHONPATH=python python -m pytest tests/nsa_prefill/test_kvcacheio_lf_pf.py -q -> 7 passed.
Not-tested: Long-running GLM5 CP+HiCache+MTP ETE throughput and host-pressure soak.
CP HiCache now treats draft KV as a strict target-owned payload through pending write visibility, host eviction, and state-buffer registration. Host metadata created before async D2H ack is no longer request-visible, so match_prefix cannot select an in-flight host node. Draft host eviction now fails before target cleanup when draft metadata is missing, and prefill/decode share one helper for draft NSA state buffers so shared-KV mode cannot silently skip mismatched draft state.
Constraint: CP shared KV + HiCache + EAGLE/MTP must not expose target-only host hits or skipped draft state as valid cache hits
Rejected: Rely on event-loop ordering and lock_ref to hide in-flight writes | match_prefix does not consult lock_ref and can observe host_len/cp_hicache directly
Rejected: Keep draft state mismatch as debug-only skip | it can poison speculative acceptance while looking like a successful cache hit
Confidence: high
Scope-risk: moderate
Directive: Do not reintroduce silent draft/target fallback in CP shared-KV HiCache paths; malformed strong-sync metadata should fail fast
Tested: python -m py_compile targeted modified files
Tested: remote g0034 container pytest test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/disaggregation/test_cp_shared_kv_transfer_mapping.py -q (91 passed)
Not-tested: Full CP shared KV + HiCache + EAGLE/MTP ETE server run after this commit
CP HiCache host hits must not advertise target residency unless the draft payload is also valid when EAGLE/MTP draft HiCache is attached. This closes target-only metadata paths by making the CP host-valid predicate and load replay fail fast, resets draft host storage with the target host pool, and records the P1-P3 strong-sync plan state.
The page-index validator is restored for CPU/fake-test tensors only, preserving unit-test coverage for malformed page spans without reintroducing CUDA hot-path host sync.
Constraint: CP shared KV + HiCache + EAGLE/MTP cannot safely demote malformed target/draft metadata to an ordinary cache miss
Rejected: keep permissive fallback for missing draft_host_indices | it can look like a successful cache hit while poisoning speculative acceptance
Rejected: re-enable generic CUDA tensor page validation | it can force host sync in the HiCache transfer hot path
Confidence: high
Scope-risk: moderate
Reversibility: clean
Directive: Do not add silent fallback around CP draft HiCache metadata; unexpected target/draft divergence should fail fast with node/rank context
Tested: remote container targeted tests: 5 passed
Tested: remote container files test_cp_hicache_metadata.py and test_hicache_controller_cp.py: 77 passed
Tested: remote container test_page_index_utils.py: 8 passed
Tested: local git diff --check and py_compile for modified Python files
Not-tested: full CP shared KV + HiCache + EAGLE/MTP ETE
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP HiCache previously let the target host pool and the draft/MTP host pool each consume the full --hicache-size budget. With EAGLE/MTP enabled this doubled per-rank host allocation and could kill scheduler ranks during startup before Python emitted a traceback. The cache now treats target KV and draft KV as one logical host-cache object: target and draft capacities are computed from one per-rank byte budget, draft may receive more token capacity when its per-token footprint is smaller, and draft attachment remains tied to target residency.
Constraint: --hicache-size is a per-rank host budget and must not be multiplied by attaching draft KV.
Rejected: Give draft another independent --hicache-size allocation | repeats the observed host OOM failure mode.
Rejected: Disable draft HiCache attachment under CP | avoids OOM but breaks target/draft cache-hit consistency for MTP.
Confidence: medium
Scope-risk: moderate
Directive: Keep target and draft KV as one logical HiCache object; do not let draft host allocation consume an independent full hicache-size budget.
Tested: python -m py_compile on modified scheduler/cache/test files
Tested: remote g0034 container PYTHONPATH=python python -m pytest test/registered/unit/mem_cache/test_cp_hicache_metadata.py -q (45 passed)
Not-tested: full multi-rank GLM5 server restart after clearing existing remote router/defunct process state
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared KV already registered the draft model main KV buffer with the
prefill/decode Mooncake managers, but NSA draft state buffers were not part
of the state registration set. HiCache/cache-hit traffic could then transfer
pages from the draft pool without transferring the matching draft
index/scale state, which is a plausible cause of the EAGLE/MTP accept-length
collapse after cache hits.
This appends compatible draft NSA state buffers to the existing state
transfer registration on both prefill and decode, and extends transfer-side
diagnostics so source/destination state-buffer counts are visible. The
mismatch guard degrades to the common prefix of registered state buffers
instead of crashing if a rolling deployment exposes asymmetric registration.
Constraint: Scope is intentionally limited to target_state_type=nsa and draft_state_type=nsa.
Rejected: Treat draft main KV transfer as sufficient | NSA attention also needs draft index/scale state for transferred pages.
Rejected: Add Mamba/SWA draft-state semantics now | those state layouts need separate correctness analysis.
Confidence: medium
Scope-risk: moderate
Directive: Do not remove the draft_state_buffer_start/count fields without checking Mooncake source/destination registration symmetry.
Tested: PYTHONDONTWRITEBYTECODE=1 python3 -m py_compile python/sglang/srt/disaggregation/prefill.py python/sglang/srt/disaggregation/decode.py python/sglang/srt/disaggregation/mooncake/conn.py
Tested: git diff --check
Tested: Remote prefill log showed registered_state_bufs=79 and maybe_send_extra_state src_state_bufs=79 dst_state_bufs=79 with no state-buffer mismatch.
Not-tested: Full accept-length recovery; latest remote run hit an unrelated prefill KV allocator idle-check leak after transfer registration succeeded.
Prefill CP only needs the local hidden shard for DeepSeek NextN draft extend. The change adds a draft shared-KV path that captures target hidden locally, feeds only the CP-local slice into the draft model, and keeps draft KV writes/transfers on the same shared logical-to-physical page mapping as target KV.\n\nDebug logs are gated behind SGLANG_CP_DRAFT_SHARED_KV_DEBUG and cover scheduler pool selection, KV manager buffer registration, local physical writes, prefill sender filtering, transfer pages, and decode commit metadata so ETE runs can prove draft KV is sharded rather than full-concatenated on a prefill rank.\n\nConstraint: Prefill runs CP while decode remains DP, so prefill must avoid full hidden/KV materialization but decode still receives full logical KV pages.\nRejected: Keep draft extend on full hidden state | preserves correctness but wastes prefill memory and defeats CP shared-KV intent.\nRejected: Transfer draft KV with a separate mapping | target and draft pools share req_to_token logical indices, so duplicating mapping adds risk without benefit.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not remove the debug logs until ETE evidence confirms draft MLA/index writes and transfer pages are CP-sharded on all ranks.\nTested: Remote compileall for changed CP draft, transfer, scheduler, NSA index, MLA write, and EAGLE files.\nNot-tested: Full GLM-5 EAGLE ETE with SGLANG_CP_DRAFT_SHARED_KV_DEBUG=1 after this logging addition; local pytest intentionally not run.
The CP shared-KV path now has a gated tai-kernel replacement for NSA index
K/scale plus MQA range preparation, and Phase8 prefetch can skip tiny prefixes
that do not cover all CP lanes. The Phase9 plan documents the next scheduler
work for overlapping CP communication with peer-request attention windows.
Temporary diagnostic logs added while validating prefetch ownership and fused
index prepare routing were removed before committing so the runtime path does
not add log-only synchronization, log counters, or shape-reporting overhead.
Constraint: Production profiling showed small per-request CPU/GPU overhead from diagnostic logging and sync-prone debug counters.
Rejected: Keep fused-index prepare fallback/used logs behind a new env var | it leaves another runtime branch and logging surface for a path that should be benchmarked with profiler evidence instead.
Rejected: Keep owned page-count prefetch logs | they require sync-prone tensor reductions and were only useful for one-off diagnosis.
Confidence: medium
Scope-risk: moderate
Directive: Reintroduce CP shared-KV diagnostics only behind explicit debug paths, and avoid .item()/shape-heavy logging in hot prefill paths.
Tested: git diff --check for staged sglang-dev changes.
Tested: AST parse for environ.py, cp_shared_kv_prefetch.py, cp_shared_kv_runtime.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py.
Not-tested: Full unit test suite.
Not-tested: Multi-node GLM5 prefill/decode/router runtime after this exact commit.
CP shared KV materialization repeatedly rebuilt the same logical-page slot remaps and page inverse metadata for each layer. Cache the token and paged remap metadata on the forward batch so MLA KV, index K/scale, and prefetch paths can reuse the layer-independent mapping while still materializing layer-specific data through the existing tai/torch runtime paths.
Constraint: Only mapping metadata is batch-scoped; dense KV/index contents remain layer-specific and are not reused.
Rejected: Cache fully materialized dense KV/index buffers | would add large per-layer memory residency and invalidation complexity.
Confidence: medium
Scope-risk: moderate
Directive: Do not assume this removes materialize or CP all-reduce cost; profile tai fallback logs and Nsight kernels before attributing E2E gains or losses.
Tested: git diff --check
Tested: remote g0034 container PYTHONPATH=python python3 -m pytest test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -q (52 passed, 5 warnings)
Not-tested: Full GLM-5 disaggregated E2E performance run
The shared-KV path now keeps more CP metadata on-device and reuses
physical out-cache locations across MLA and NSA index writes, so each
layer avoids repeating logical-to-physical remaps. The in-seq CP
all-gather rerange path now delegates to tai-kernel when available and
falls back to the existing torch split/cat path with an explicit log.
This also extends the Phase8 prefetch machinery to cover shared KV
materialization metadata and keeps debug/fallback behavior gated so the
fast path is not polluted by diagnostic checks.
Constraint: Custom CP kernels must live in tai-kernel and be imported lazily from SGLang
Constraint: Decode does not use CP; these changes target NSA prefill CP in-seq-split shared KV
Rejected: Recompute physical local cache locations separately for MLA and index writes | repeats the same remap work every layer
Rejected: Keep the in-seq rerange Triton code inline in SGLang | duplicates kernel ownership and blocks tai-kernel reuse
Confidence: medium
Scope-risk: moderate
Directive: Keep CP collective ordering identical across ranks; do not add rank-local fallback decisions inside shared KV materialize paths
Tested: Remote g0034 container py_compile for modified SGLang/tai-kernel files; remote pytest test/registered/unit/layers/test_nsa_cp_utils.py passed with 24 tests
Not-tested: Full multi-node GLM5 prefill/decode throughput after the final commit boundary
The shared-KV prefill path now optionally calls tai_kernel.nsa_prefill.fused_store_mla_kv before falling back to logical_locs_to_physical plus set_mla_kv_buffer. The fast path supports packed FP8 and BF16/FP16 direct KV buffers, while debug mode and kernel failures still preserve the existing fallback behavior. Success logging was removed after path verification because per-layer/per-rank logs are too noisy in normal server runs.
Constraint: Runtime must remain safe when tai-kernel is absent or debug checks are enabled
Rejected: Keep success logs permanently | floods prefill logs once every rank/layer starts using the fast path
Confidence: high
Scope-risk: moderate
Directive: Keep fallback warnings; do not re-add per-layer success logs outside explicit debug instrumentation
Tested: g0034 container python -m py_compile python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py
Tested: g0034 container PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -q (40 passed)
Not-tested: Full multi-node PD server throughput after log removal
CP shared KV now avoids the PyTorch sort/search remap for the single-request current-only path by deriving compact rows from page-level inverse mapping. The same change keeps sort NVTX attribution gated and splits high-frequency MoE sort markers behind a separate env var so profiling does not perturb normal runs.
Decode-side disaggregation prealloc also avoids rebuilding large token index tensors and records finer allocation timing, while compute-owner allocation/free tests cover the shared-KV page-lane behavior.
Constraint: The runtime tree used for validation is the remote /sgl-workspace/sglang-tai mount, which is not itself a Git repository, so these tracked files were synchronized into the local repo before commit.
Rejected: Keep torch.sort/searchsorted for current remap | it emits ATen/CCCL radixSortKVInPlace kernels in the attention hot path.
Rejected: Enable MoE sort NVTX under the generic sort env | the MoE preprocess sort is too frequent and can make profiling look like a hang.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce token-level torch.sort/searchsorted in CP shared-KV current remap without profiling the attention hot path under Nsight.
Tested: Remote container py_compile for modified runtime files; git diff --cached --check.
Not-tested: Full multi-node GLM5 PD throughput/profile rerun after the page-inverse current remap.
Phase8 only gains useful overlap when the next-layer MLA prefix prefetch is allowed to run until the next layer actually consumes the prefetched buffer. The old wait-after-attention switch let runtime configuration collapse the optimization back into current-layer tail latency, so the prefetch path now has one wait policy and the documentation records the implemented behavior.
Constraint: Phase8 should keep the production environment surface minimal while preserving the existing enable and debug-log knobs
Rejected: SGLANG_CP_SHARED_KV_MLA_PREFETCH_WAIT_AFTER_ATTENTION | it reintroduced current-layer synchronous waiting and made profiling behavior depend on a nonessential policy knob
Confidence: medium
Scope-risk: narrow
Directive: Do not add another Phase8 wait policy knob without first proving the added policy improves end-to-end prefill latency under CP shared KV
Tested: Python AST parse for touched Python files
Tested: git diff --check
Not-tested: Full pytest and remote server integration were not run in this commit
