Tiny extend requests can leave most CP lanes without query work, which has been tied to hangs and accept-length regressions. This change introduces a dual valid/compute metadata contract: forward paths may materialize compute-padded rows, while cache, current reuse, direct write, HiCache backup, and load remain valid/page based.
The implementation keeps radix/HiCache/device allocation on real page extents, filters dummy compute rows before MLA/index cache writes and current reuse, makes top-k/index consume compute rows while compacting valid rows, and opens tiny CP shared-KV in-seq split through compute padding. The accompanying plan document records the contract and P1-P7 evidence.
Constraint: CP shared KV and HiCache must stay page-granular; dummy compute rows must not allocate, write, backup, or load KV cache.
Constraint: Avoid silent fallback and avoid adding collectives on hot paths.
Rejected: Pad cache allocations to cp_size pages | would waste KV capacity and pollute radix/HiCache state.
Rejected: Keep tiny suffixes out of CP split | preserves the zero-lane behavior that compute padding is meant to remove.
Confidence: medium
Scope-risk: broad
Directive: Do not route compute-padded dummy rows into out_cache_loc, current reuse, HiCache reservation, or backup descriptors; keep valid/cache metadata explicit.
Tested: Remote g0034 container targeted P7 tests: 3 passed, 3 warnings.
Tested: Remote g0034 container full unit slice: PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/mem_cache/test_cp_shared_kv_layout.py => 214 passed, 5 warnings, 2 subtests passed.
Tested: Local py_compile for touched P7 test file.
Not-tested: Latest CUDA/ETE traffic validation for dummy top-k rows, accept len, output len, and detokenizer hang behavior.
CP HiCache write reservations must stay per radix node, but the transfer descriptor does not need to be per request. This changes the layer-end hook to group pending write states for the same source and layer, so bs>1 prefill emits one target D2H descriptor and one draft D2H descriptor per layer while preserving per-node metadata, rollback, and ack semantics.\n\nConstraint: CP shared-KV HiCache metadata, host slots, and radix acknowledgements remain per request/node.\nConstraint: TAI direct transfer kernels already accept flattened page descriptors, so no tai-kernel change is required.\nRejected: Merge HiCache reservations or radix nodes | would complicate rollback and split handling.\nRejected: Add collective synchronization for grouped backup | grouping is local descriptor construction and must not add rank-level sync.\nConfidence: high\nScope-risk: moderate\nDirective: Keep target and draft source notifications separate; final ack must wait for both when draft HiCache is attached.\nTested: local py_compile for cache_controller.py and test_hicache_controller_cp.py\nTested: local git diff --check\nTested: remote pytest test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py => 85 passed, 3 warnings\nNot-tested: full ETE bs>1 CP HiCache replay with admission gate removed\nNot-tested: Nsight/throughput validation of reduced D2H submit count
The bs>1 path needs index top-k, shared-index prepare, current-index compact, and current-slot compose to consume flattened batch descriptors instead of falling back to per-request or per-segment Python/Torch work. This change wires SGLang to the new TAI batch prepare kernels, keeps fallback explicit, and records the remaining HiCache/load-backup gaps in the bs>1 workstream docs.
Constraint: CP shared-KV bs>1 must reuse fast paths rather than adding slow batch-only fallbacks
Constraint: No new collective operations were introduced
Rejected: Leave current-only cp_index as Python slice/cat | it keeps per-segment overhead in the short-extend bs>1 case
Rejected: Infer max segment lengths from CUDA descriptor tensors | .item() would add CPU synchronization on the hot path
Confidence: medium
Scope-risk: moderate
Directive: Do not remove the explicit fallback warnings without verifying the corresponding TAI symbols are present in production
Tested: local py_compile for touched SGLang files
Tested: remote g0034 test_nsa_cp_utils.py passed, 53 tests
Tested: remote g0034 test_fill_current_index_page_slots_uses_tai_kernel_when_available passed
Not-tested: full ETE bs>1 traffic with HiCache load/backup and draft/EAGLE enabled
PR #10 provides the W2 owner-lane allocation side needed by the current bs>1 CP shared-KV work: supported multi-request extends now derive page owners per request, preserve flattened request order, and use owner-lane allocation instead of legacy page allocation.\n\nThe merge applies cleanly on top of the W3/W4 target current-reuse work because the touched runtime surface is allocator/mem_cache only. Focused remote tests cover the new owner-lane planner/allocation behavior and the existing W4 target reuse regressions.\n\nConstraint: CP shared-KV page ownership is request-relative and page-granular.\nConstraint: Supported bs>1 CP shared-KV allocation must not silently fall back to legacy allocation.\nRejected: Treat multiple requests as one concatenated extend | would assign owners relative to the wrong request boundary.\nRejected: Keep multi_batch fallback | breaks downstream direct-write owner assumptions.\nConfidence: medium\nScope-risk: moderate\nDirective: Preserve request-order flattening when changing owner allocation; do not reorder by lane for convenience.\nTested: Local py_compile for allocator.py, common.py, cp_shared_kv_compute_owner.py, and test_cp_shared_kv_layout.py.\nTested: Local git diff --check --cached.\nTested: Remote g0034 py_compile for touched mem_cache files and test_cp_shared_kv_layout.py.\nTested: Remote g0034 pytest test_cp_shared_kv_layout.py => 34 passed, 3 warnings.\nTested: Remote g0034 pytest test_nsa_cp_utils.py test_cp_shared_kv_runtime.py => 157 passed, 5 warnings, 2 subtests passed.\nNot-tested: Full ETE/perf run; production HiCache load/back up with bs>1 allocation under sustained traffic.
Batch-size > 1 cache-hit traffic must not lose the current/partial-current reuse fast path. This change extends the target MLA/index sync-correct path to validate batched current suffix rows, compose page-aligned prefix spans, and route batched index top-k through current-only or partial-current reuse without falling back to scalar guards.\n\nThe implementation keeps page as the minimum cache unit: prefix cache coverage is page-aligned, while current suffix rows are sliced by valid extend lengths so padded tail rows are not exposed to attention. The index top-k batch path now mirrors the single-request contract: current-only reuses current index KV directly, partial-current materializes once and shares the dense buffer across request segments.\n\nConstraint: CP shared-KV supports in-seq zigzag only; round-robin remains fail-fast for shared-KV.\nConstraint: No new collectives are introduced; this is a sync correctness path, not a new communication scheme.\nRejected: Keep bs>1 current_index_kv fail-fast | disables the cache-hit path that W4 is meant to restore.\nRejected: Pad requests to a common batch length | violates the page-granular contract and exposes padded tails to consumers.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce batch_size_not_one or batch_gt1 current-reuse guards without proving an equivalent batched fast path exists.\nTested: Local py_compile for touched runtime/test files.\nTested: Remote g0034 pytest test_nsa_cp_utils.py test_cp_shared_kv_runtime.py => 157 passed, 5 warnings, 2 subtests passed.\nTested: Remote g0034 pytest test_cp_shared_kv_layout.py => 27 passed, 3 warnings.\nNot-tested: Full ETE with live traffic; CUDA kernel-backed batched descriptors; L1 prefetch, HiCache load/backup, and draft/EAGLE bs>1 current reuse.
Route supported multi-request CP shared-KV extend allocation through request-local owner plans so downstream direct-write paths can trust flattened out_cache_loc owner lanes. Unsupported page plans now fail fast instead of falling back to legacy allocation.
Include the normative govctl RFC and completed work-item trail in the tracked repository; keep transient superpowers planning outside the sglang branch.
Constraint: RFC-0001 W2 requires no legacy multi_batch fallback for supported bs>1 owner-lane allocation
Rejected: concatenating requests before owner planning | CP ownership is request-relative
Rejected: committing only docs/superpowers plan | it omits the normative RFC and work-item trail
Confidence: high
Scope-risk: moderate
Directive: Preserve flattened request order when changing CP shared-KV allocation; run govctl from the sglang repository root
Tested: remote uv test_nsa_cp_utils.py 39 OK; remote uv test_cp_shared_kv_layout.py 34 OK; remote uv test_alloc_pages_with_owners.py 10 OK; govctl check; git diff --check
Not-tested: CUDA ETE/perf paths beyond focused W1/W2 unit suites
W4-1 needs target index/top-k sync correctness before current/partial-current reuse can be restored. Batch-size>1 in-seq CP produces local q/weights in request-segment order, so top-k must consume req0_prev, req0_next, req1_prev, req1_next rather than treating the flattened batch as one scalar prev/next pair.
The implementation adds a batch dispatch for _get_topk_in_seq_cp_pair, reuses one synchronous shared-index materialization per layer, and calls _get_topk_ragged_with_cp per request segment with an explicit batch_idx. The scalar bs=1 path remains unchanged.
Constraint: This is W4-1 target index/top-k sync correctness; original W4 current/partial-current reuse remains a separate follow-up.
Constraint: Phase W4-1 must not enable bs>1 index prefetch, current reuse, partial-current reuse, or the cp_index multi-batch branch.
Rejected: Use cp_index branch for multi-batch | source marks that path as having accuracy issues.
Rejected: Pad batch requests to max length | wastes compute and violates packed/ragged batch contract.
Confidence: high
Scope-risk: moderate
Directive: Keep bs>1 target top-k ordered by request segment unless a later fused descriptor proves identical ordering and correctness.
Tested: Remote g0034 py_compile for nsa_indexer.py
Tested: Remote g0034 PYTHONPATH=python pytest test/registered/unit/layers/test_nsa_cp_utils.py -> 45 passed
Tested: Remote g0034 PYTHONPATH=python pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py -> 172 passed, 2 subtests passed
Not-tested: Full ETE bs>1 serving run with live traffic
W3 needs batch-size>1 extends to use packed valid tokens while preserving per-request page boundaries. The local out_cache_loc planner now validates against the batch plan's request lengths instead of the first request's scalar split metadata, then reuses the existing batch split helper to produce this rank's logical/physical cache locs.
Direct-write failures inside the CP shared-KV contract now fail fast instead of silently falling back to legacy index/MLA stores. This exposes allocator, owner-lane, page-alignment, and shape-contract bugs early for both bs=1 and bs>1.
Constraint: bs>1 batching must not pad short requests to the longest request; only per-request page-boundary padding is allowed.
Rejected: Keep bs=1 compatibility fallback | it hides CP shared-KV contract violations and caused repeated slow-path ambiguity.
Rejected: Pad batch to max request length | wastes compute and complicates cache validity for short extends.
Confidence: high
Scope-risk: moderate
Directive: CP shared-KV contract errors should stay fail-fast; do not reintroduce silent direct-write fallback without ETE evidence and explicit warning semantics.
Tested: Remote g0034 py_compile for utils.py nsa_indexer.py forward_mla.py
Tested: Remote g0034 PYTHONPATH=python pytest test/registered/unit/layers/test_nsa_cp_utils.py -> 43 passed
Tested: Remote g0034 PYTHONPATH=python pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py -> 170 passed, 2 subtests passed
Not-tested: Full ETE bs>1 serving run with live traffic
Batch-size support needs request-first CP metadata; treating a batch as one long sequence breaks page ownership, top-k ranges, and phase1 compact output collection. This adds a batch CP plan that records per-request page-aligned splits, rank-local offsets, kv/actual-seq metadata, last-token owners, and flattened descriptors for downstream allocator/runtime workstreams.
The scalar full-rerange path now fail-fasts for batch metadata so bs>1 cannot silently discard the narrow-output optimization or restore hidden states with single-request assumptions.
Constraint: CP shared-KV cache state is page-owned and must preserve request boundaries under bs>1.
Rejected: Let bs>1 fall back to scalar full hidden rerange | it loses the phase1 communication reduction and uses wrong single-request metadata.
Rejected: Add a collective to confirm batch plans | all ranks can derive the same plan from CPU metadata and config.
Confidence: medium
Scope-risk: moderate
Directive: Do not remove batch fail-fast guards until W2/W3 consumers use CPSharedKVBatchPlan end-to-end.
Tested: python -m py_compile python/sglang/srt/layers/attention/nsa/utils.py test/registered/unit/layers/test_nsa_cp_utils.py
Tested: remote g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py -> 39 passed
Not-tested: full ETE bs>1 CP shared-KV runtime; W2/W3 allocator/direct-write consumers are not implemented yet
The missing index prefetcher state is expected when MLA/index prefetch is disabled. Logging it as an index-prefetch fallback made the normal synchronous materialize path look like a failed fast path and created noisy reports.
This keeps consume-miss warnings for enabled prefetchers, but suppresses missing-prefetcher warnings when the prefetch feature gate is off.
Constraint: Index prefetch is controlled by the same MLA prefetch enable gate in the current pipeline.
Rejected: Suppress all index prefetch fallback warnings | consume misses still indicate an enabled prefetch pipeline failed to serve a requested buffer.
Confidence: high
Scope-risk: narrow
Directive: Missing prefetcher is not a fallback when SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH=0; do not re-add warning noise without a separate enable gate.
Tested: Local py_compile for nsa_indexer.py and test_cp_shared_kv_runtime.py.
Tested: Remote g0034 targeted regression set passed 4 tests: disabled sync materialize no warning, consume miss warning, first-layer no-warning, enabled create-skip warning.
Not-tested: Full CUDA ETE run with logs.
Host HiCache reservations were paying token-level free-slot scans when trying to preserve page contiguity. The allocator now keeps a lazy page-extent index so availability checks and contiguous-preferred allocations avoid materializing the full 220GB-equivalent free-slot metadata path.
The companion benchmark models steady-state L2 churn near full occupancy, including burn-in and historical node-size effects, so LPF/RDMA descriptor quality can be separated from ETE noise.
Constraint: CP HiCache host allocations are page-shaped, but existing callers may still read free_slots directly.
Rejected: Sort and scan free_slots on each alloc_contiguous_preferred call | measured ms-level CPU overhead on 220GB-equivalent metadata.
Rejected: Remove free_slots compatibility | storage/tests still rely on the public tensor surface.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce per-allocation full free_slots scans on HostKVCache; preserve page-extent metadata or benchmark before changing allocator shape.
Tested: Local py_compile for memory_pool_host.py, allocator benchmark, and related tests.
Tested: Local test_cp_hicache_allocator_bench.py 10 passed.
Tested: Remote g0034 test_hicache_controller_cp.py 67 passed; test_cp_hicache_allocator_bench.py 10 passed.
Tested: Remote 220GB-equivalent host_churn benchmark: contiguous path reduced from ms-level to ~30-292us p50 depending on fragmentation.
Not-tested: Full CUDA ETE run after allocator change.
Not-tested: Production long-run fragmentation behavior under live traffic.
The CP shared-KV allocator was still doing total-cache-sized CPU work in the scheduler hot path. That cannot be hidden by GPU overlap, so owner-lane allocation now maintains per-owner free/release buckets and consumes request-sized prefixes instead of rebuilding masks over the full free-page tensor on each request.\n\nThe benchmark was extended to isolate L1 stats, selection, and allocation costs, and the CPU layout tests now install a complete sgl_kernel stub before importing SGLang helpers so remote unit collection does not abort in native extension loading.\n\nConstraint: Allocator CPU work blocks scheduler progress and cannot overlap with GPU forward execution.\nConstraint: CPU unit tests must not load native sgl_kernel on remote images where the loader can SIGABRT.\nRejected: Keep contiguous-run search over full free_pages | still scales with cache capacity and measured multi-ms overhead.\nRejected: Treat remote collection abort as an environment-only issue | it prevented allocator regression coverage and was fixable with a test-local stub.\nConfidence: high\nScope-risk: moderate\nDirective: CP owner-lane allocation is bucket-based; do not reintroduce full free_pages scans on the hot path without benchmark evidence.\nTested: Local py_compile for touched files\nTested: Local benchmark unit test, 6 passed\nTested: Remote benchmark unit test, 6 passed\nTested: Remote test_alloc_pages_with_owners.py, 10 passed\nTested: Remote test_cp_shared_kv_layout.py, 27 passed\nTested: Remote production allocator microbench shows select/alloc p50 reduced from ms-scale to sub-ms scale\nNot-tested: Full ETE traffic run after allocator bucket change
CP shared KV now keeps explicit L1 and host free-room targets so pressure is handled by planned eviction instead of repeated capacity-edge retries. The host allocator gains contiguous-preferred page reservation, L1 owner-lane allocation prefers contiguous physical pages, and CP HiCache metadata preserves pending backup safety for page-granular radix updates. Mooncake transfer stats and allocator microbenchmarks are included to make the remaining transfer bottlenecks measurable rather than inferred.
Constraint: CP shared KV uses decode CP size 1 with all prefill CP ranks participating in transfer, so L1/L2 cache residency must remain page-granular and avoid extra collectives.\nConstraint: Production HiCache can be hundreds of GB, so allocator metadata overhead must be visible before enabling aggressive contiguous allocation broadly.\nRejected: Evict only the exact deficit | this keeps the cache at the cliff and causes repeated evict/allocate pressure.\nRejected: Rely on allocator scans alone for contiguity | remote microbenchmarks show fragmented 220GB-equivalent host metadata can make contiguous-preferred scans multi-ms.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not increase L1/L2 free-room defaults or add new CP collectives without ETE evidence and transfer/allocator measurements.\nTested: python -m py_compile on touched runtime/test/benchmark files.\nTested: PYTHONPATH=. python -m pytest -q test/registered/unit/benchmark/test_cp_hicache_allocator_bench.py => 4 passed, 1 warning.\nTested: Remote g0034 log /mnt/beegfs/cjy/log/sglang_cp_hicache_20260601_233723.log shows active prefill process with L1/L2 free-room args, 702 HTTP 200 chat completions, 6272 prefill batches, and no fatal scheduler traceback in latest scan.\nTested: User-reported L1/L2 cache ETE validation passed on remote run.\nNot-tested: Full local pytest suite; local environment is missing several runtime dependencies.\nNot-tested: CUDA allocator microbenchmark during active production prefill process.\nNot-tested: Mooncake straggler fix; stats show transfer tail latency remains a separate bottleneck.
CP shared-KV performance debugging depends on seeing when the runtime leaves the intended TAI, IPC, prefetch, or current-reuse paths. This change makes those misses visible through standardized warning markers while keeping per-reason log limits to avoid per-layer log floods.\n\nThe warnings intentionally distinguish fallback from fail-fast: unsupported correctness-sensitive states still raise, while performance-path misses emit [CP_SHARED_KV_FALLBACK] with the concrete reason.\n\nConstraint: Production ETE debugging needs visible fallback evidence without enabling heavy debug mode, which can itself disable fast paths.\nRejected: Rely only on optional MLA prefetch debug logs | they are env-gated, layer-limited, and miss non-prefetch TAI/IPC/current-reuse fallbacks.\nRejected: Log every per-layer event without limits | would drown useful transfer/cache diagnostics under steady-state traffic.\nConfidence: high\nScope-risk: moderate\nDirective: Do not remove these fallback warnings unless an equivalent low-noise observability path exists for every fast-path miss.\nTested: local py_compile for touched files; local git diff --check for touched files; remote g0034 py_compile and pytest for test_nsa_cp_utils.py, test_cp_shared_kv_layout.py, test_cp_shared_kv_runtime.py passed before commit (156 passed, 5 warnings, 2 subtests passed).\nNot-tested: full ETE serving traffic after warning additions.
Chunked prefill was skipping prepared CP HiCache backup entirely while still inserting unfinished valid-tail radix state. That combination caused middle chunks to miss reusable host cache and repeatedly hit stale-tail split fallback paths.\n\nThis keeps CP HiCache page-granular: middle chunks now reserve/write only completed pages, and unfinished radix inserts expose only page-complete prefixes while preserving the request-owned sub-page tail in prefix_indices for later chunks.\n\nConstraint: CP HiCache radix and host residency are managed at page granularity, but chunked prefill can end on a sub-page tail.\nRejected: Continue skipping chunked backup | loses middle-chunk reuse and triggers fallback storms.\nRejected: Insert sub-page chunk tails into CP HiCache | creates stale-tail prune/split conflicts against request-owned KV.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce chunked_req backup skip; keep CP HiCache chunked inserts page-complete unless the radix/host cache contract becomes sub-page aware.\nTested: python -m py_compile python/sglang/srt/mem_cache/radix_cache.py python/sglang/srt/mem_cache/hiradix_cache.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py\nTested: g0034 targeted chunked CP HiCache regression tests, 2 passed\nTested: g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py, 107 passed\nNot-tested: Fresh chunked-prefill ETE traffic run after process restart
Prepared per-layer CP HiCache backups can enqueue their final write ack before the radix node is attached. Chunked prefill exposed this window when a separate catch-up backup made writing_check drain the ack queue and pop an unattached node id.
Keep ready but unattached prepared acks queued until insert either attaches the prepared backup or rollback removes the orphan ack. Also document the reactive host free-room eviction plan separately from this state-machine fix.
Constraint: CP HiCache prepared backup transfer can complete before radix insertion attaches node state
Rejected: Drop unknown ack ids | would orphan a later successful prepared attach and leak write state
Rejected: Chunked-only guard | the invalid assumption is in the generic CP write ack state machine
Confidence: high
Scope-risk: narrow
Directive: Do not drain CP write acks unless every ack id is registered in pending_host_backups or ongoing_write_through
Tested: Remote red-green test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_writing_check_defers_unattached_prepared_ack
Tested: Remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py (116 passed)
Tested: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py
Not-tested: Full chunked-prefill ETE replay after this commit
Co-authored-by: OmX <omx@oh-my-codex.dev>
Chunked prefill can revisit a sub-page CP HiCache tail while a per-layer backup is still in flight. The old insert path split the radix node first and only then tried to prune the stale tail, so an unprunable pending backup raised after tree mutation and propagated to the scheduler.
This makes split/prune atomic from the radix-tree perspective: drain completed write acks only on the conflict path, preflight pending/unprunable state before split, and return a deferred insert result when the backup is still in flight. Unfinished requests keep their KV ownership for transfer/release instead of rematching or freeing pages under a stale tree state.
Constraint: CP HiCache backup metadata is node/page owned and cannot be repartitioned while per-layer D2H is pending
Rejected: Split the pending backup node | would require repartitioning in-flight backup metadata and host reservations
Rejected: Delete the stale tail unconditionally | risks freeing device/host pages still owned by pending backup
Confidence: high
Scope-risk: moderate
Directive: Do not move stale-tail prune after split without a preflight; pending backup split conflicts must remain non-mutating
Tested: remote g0034 container py_compile for mem_cache files
Tested: remote g0034 PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py => 115 passed
Tested: local git diff --check
Not-tested: full chunked prefill ETE after service restart
Co-authored-by: OmX <omx@oh-my-codex.dev>
NSA FP8 CP shared-KV reuse must operate on packed page-slot rows, not bf16 compact rows. The change keeps current-only and partial-current reuse inside the page-aligned materialization contract, fails fast for non-page-aligned CP split inputs, and prevents FP8 FlashMLA-KV prefill from reaching incompatible in-seq CP metadata.
Constraint: NSA FP8 persistent MLA KV rows are packed 656-byte records and CP shared KV cache management is page-granular.\nConstraint: FlashMLA-KV prefill metadata is not CP-local after NSA in-seq splitting.\nRejected: Silently splice bf16 current rows into FP8 materialized cache | corrupts the packed cache layout.\nRejected: Keep FP8 FlashMLA-KV auto prefill under NSA CP | reaches num_splits shape errors after q-row splitting.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable FP8 FlashMLA-KV prefill for NSA in-seq CP until metadata is rebuilt after CP splitting or made CP-local.\nTested: Local git diff --check and py_compile for touched SGLang files.\nTested: Remote g0034 related unit sweep recorded in docs: test_nsa_cp_utils.py, test_cp_shared_kv_layout.py, test_cp_shared_kv_runtime.py, test_cp_hicache_metadata.py passed.\nNot-tested: Full FP8 ETE startup and performance run after this commit.
Launching CP shared-KV prefetch from MLA prepare or the indexer can make
next-layer prefix work overlap current-layer MQA/materialization instead of
the attention window. Centralize the launch in the NSA backend after
current-layer materialization and before attention, and leave the early
indexer hook inert so the call site cannot regress silently.
The accompanying notes capture the draft-as-forward-layer follow-up plan and
the latest OOM diagnosis: the observed 178945-token failure matches the CP
in-seq MQA logits allocation, so the follow-up fix is q-dimension chunking in
_get_topk_ragged_with_cp(), not a max-prefetch-size gate.
Constraint: CP shared-KV prefetch must avoid overlapping current-layer MQA/materialization and must not add silent fallback behavior.
Rejected: Limit maximum prefetch size | hides the CP logits peak and can reduce cache/prefetch effectiveness.
Rejected: Keep prepare/indexer launch sites | they place next-layer collectives in the wrong overlap window.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce early CP prefetch launch without checking Nsight overlap and CP MQA memory peaks.
Tested: Local git diff --check and py_compile for touched Python files.
Tested: Remote container py_compile plus targeted pytest: 3 passed, 5 warnings.
Not-tested: Full ETE under production traffic after this commit.
Not-tested: CP in-seq MQA logits chunking; documented as follow-up.
Synchronous CP shared KV full-hit and partial-current paths can now use the tai-kernel CUDA IPC slot-dense materialize path instead of first copying local owner pages and then running a dense CP all-reduce. This keeps the async prefetch pipeline unchanged while routing the safer synchronous runtime path through descriptor-driven owner-page reads.
The runtime builds descriptors from the existing page-aligned slot contract, caches peer pointer tables for long-lived KV/index buffers, and falls back with explicit warnings if the tai-kernel IPC capability is missing. Unit coverage locks offset handle exchange, descriptor construction, and both MLA/index full and partial-current calls.
Constraint: Async prefetch currently has unresolved scheduling contention with GEMM/MoE and current-layer KV collectives, so this commit intentionally does not wire IPC into prefetch.
Constraint: tai-kernel must provide offset-aware CUDA IPC symbols from af9fb67.
Rejected: Replace prefetch all-reduce in this slice | Nsight shows prefetch timing and communicator contention need a separate scheduling design.
Rejected: Fail-fast on missing tai IPC immediately | remote ETE still needs to validate production deployment capability before removing the warning fallback.
Confidence: medium
Scope-risk: moderate
Directive: Keep prefetch and synchronous materialize decisions separate until prefetch has a low-SM/copy-engine schedule.
Related: tai-kernel af9fb67
Tested: git diff --check; remote runtime/unit evidence recorded in docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md.
Not-tested: Fresh GLM5 ETE after this commit; async prefetch IPC path.
Co-authored-by: OmX <omx@oh-my-codex.dev>
The replay failure returns HTTP 400 before generation, so the next diagnostic point must be the chat-template conversion boundary rather than HiCache or scheduler paths. This adds a failure-only summary that records message roles, content shapes, empty indices, tool metadata, bootstrap room, and template mode without logging raw user content.
Constraint: Production replay traffic can contain large/private messages, so diagnostics must avoid full content dumps.
Rejected: Enable full request logging | too noisy and exposes complete prompt payloads.
Rejected: Normalize empty content immediately | would hide whether the malformed message is user, assistant, tool, or template-derived.
Confidence: high
Scope-risk: narrow
Directive: Keep this log failure-only; do not move it to the hot path or print raw content.
Tested: python -m py_compile python/sglang/srt/entrypoints/openai/serving_chat.py
Tested: remote docker py_compile after scp to g0034:/mnt/beegfs/cjy/sglang-dev
Not-tested: Full replay reproduction; requires user-driven traffic after service restart.
Page-aligned CP shared KV can pad out_cache_loc beyond valid current rows, so current reuse now gates MLA composition on the valid extend rows and permits draft partial-current reuse once the TAI sparse-page capability check passes. The TAI current-slot path self-tests sparse pages before use and falls back to the torch reference when the installed kernel is stale.
Eviction success and no-op diagnostics were also moved from INFO to DEBUG so owner-lane and host-admission churn does not flood production logs; true write failures remain WARNING.
Constraint: CP shared KV uses page-aligned physical reservations where valid suffix rows can be shorter than padded out_cache_loc.
Constraint: Production failure/fallback logs must remain visible, but hot successful eviction paths should not emit INFO per victim/rank.
Rejected: Keep draft partial-current reuse disabled | would preserve avoidable full materialization on draft cache-hit suffixes.
Rejected: Trust the TAI current-slot kernel unconditionally | stale kernels can corrupt sparse current-page composition.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce INFO logging in eviction hot paths without rate limiting and runtime evidence.
Tested: local py_compile for touched Python files
Tested: local git diff --check
Tested: remote container py_compile for touched Python files
Tested: remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py::TestHiCacheEvictLoggingLevels::test_evict_hot_path_success_logs_are_debug_only test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 82 passed, 5 warnings, 2 subtests passed
Not-tested: full ETE traffic after this commit; draft partial-current accept length still needs user-driven runtime validation
The index path now mirrors the target MLA partial-current contract: prefetched or synchronously materialized prefix pages are composed with valid current index K/scale rows in slot-dense page buffers. Current-only batches keep the compact current-index fast path, while partial cache-hit batches share one composed dense index buffer across the in-seq prev/next topk pair.\n\nThe prefetch consume path remaps through the slot page inverse instead of treating the slot-dense buffer as physical-pool capacity, and current-index quantization uses valid extend rows so padded out_cache_loc does not disable reuse.\n\nConstraint: CP shared KV remains page-slot based; padding rows must stay invisible to attention/index semantics\nConstraint: Draft/EAGLE partial-current reuse remains guarded by should_reuse_current_extend_kv\nRejected: Replace prefix all-reduce with all-gather | NCCL all-gather still uses SM and would require an additional compose/scatter step\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce current-only gating for index reuse; partial target cache hits must compose prefix + valid current rows\nTested: Local py_compile for touched Python files\nTested: g0034 sglang-glm5-dev-2 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 77 passed, 5 warnings, 2 subtests passed\nNot-tested: Full ETE traffic with latest commit; CUDA perf impact of index partial-current prefetch under production load
CP HiCache direct/page_first_direct all-layer backup was still able to enter sgl-kernel's stale cudaMemcpyBatchAsync path, which segfaults under CUDA 13 before Python can surface an error. The SGLang route now avoids that all-layer sgl-kernel op for page_first_direct backup and uses the TAI per-layer direct LF->PF op for MHA, MLA, and NSA indexer data.\n\nThe load-back path also prepares NSA indexer page indices once per load op and reuses them across per-layer H2D loads, preserving per-layer overlap while removing redundant page-index derivation.\n\nConstraint: Remote runtime is CUDA 13.0 where sgl-kernel's all-layer direct LF->PF op uses the wrong cudaMemcpyBatchAsync ABI.\nRejected: Patch sgl-kernel in this branch | we are converging production HiCache direct/page_first_direct paths onto tai-kernel and do not want to maintain another CUDA-ABI-sensitive copy path here.\nRejected: Collapse H2D load-back into one all-layer op | that would reduce submit count but lose per-layer completion visibility and forward overlap.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce sgl_kernel.transfer_kv_all_layer_direct_lf_pf for direct/page_first_direct HiCache backup without CUDA 13 ABI verification.\nTested: g0034 container: PYTHONPATH=python python -m pytest -q -s test/registered/unit/mem_cache/test_nsa_pool_host_unit.py -> 10 passed, 3 warnings.\nTested: g0034 container: PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py -> 61 passed, 3 warnings.\nTested: python -m py_compile python/sglang/srt/mem_cache/memory_pool_host.py python/sglang/srt/managers/cache_controller.py test/registered/unit/mem_cache/test_nsa_pool_host_unit.py test/registered/unit/managers/test_hicache_controller_cp.py\nNot-tested: Full ETE prefill/decode traffic after this commit.\nNot-tested: sgl-kernel implementation itself remains unchanged.
The failing runs showed EAGLE accept length collapsing when draft cache-hit suffixes used the new partial-current splice path. This keeps target partial-current reuse enabled, but returns EAGLE/NextN draft cache-hit suffixes to the previous full-materialize path with an explicit fallback warning until the draft splice path has value-level ETE proof.\n\nThe same change set also tightens the page-granular CP HiCache contract for scheduler-visible hits and makes the prefill-to-decode EAGLE handoff observable without cloning hot-path metadata. Exact non-page CP hits are floored to a page boundary for new scheduling decisions, while internal unfinished-request refresh keeps its exact accounting.\n\nConstraint: CP shared KV and HiCache operate at page granularity; exposing token-precise CP tails to scheduler-visible cache hits can force non-page partial materialization.\nConstraint: EAGLE/NextN draft has only one executable layer, so draft prefetch and draft partial-current splice need a separate correctness contract from target layers.\nRejected: Keep draft partial-current splice enabled | remote logs correlate it with avg accept length around 0.068 and median 0.\nRejected: Clone decode metadata tensors on transfer | slot ownership until process_prebuilt consumes them avoids extra hot-path copies.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable draft partial-current reuse without metadata/draft-KV value checks and ETE accept-length evidence.\nTested: g0034 container py_compile for touched modules.\nTested: g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 183 passed, 5 warnings, 2 subtests passed.\nNot-tested: Fresh ETE accept-length run after this exact commit; requires user-driven traffic restart.
CP HiCache owns KV at page granularity, but exact valid-tail extension and backed partial-tail split could leave an old sub-page tail child beside a new suffix that reuses the same physical page. That makes radix residency ambiguous across device, host, and draft mirrors. The insert/match split paths now prune stale floored tails when safe, and defer/fail through the existing pending-split path when the subtree is protected or has in-flight backup state.\n\nThis also keeps a temporary scheduler boundary warning for externally observed zero-output responses so future ETE runs can classify whether zero visible output reaches SGLang's output processor.\n\nConstraint: CP shared KV and HiCache manage physical KV by page, while radix keys retain valid-token lengths.\nRejected: Keep overlapping old tail nodes after page-floor split | leaves two independent cache states for one physical tail page.\nRejected: Force-prune protected or in-flight backup tails | can mutate cache state still used by active transfer or inference.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not remove the stale-tail prune without replacing it with another page-granular ownership rule for CP HiCache radix splits.\nTested: Remote py_compile for hiradix_cache.py and scheduler_output_processor_mixin.py in g0034 container.\nTested: Remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py -> 97 passed, 5 warnings.\nNot-tested: Full ETE recovery of EAGLE accept length; latest ETE still shows accept collapse, documented in C55.\nNot-tested: Router/client-side output_len=0 correlation when server-side OUTPUT_ZERO_DEBUG does not fire.
Transferred EAGLE metadata buffers are reusable slot views. The previous clone-based mitigation protected correctness but added copies on the transfer hot path and hid the actual lifetime contract. This change makes the transferred request own the metadata slot while it sits in the decode waiting queue, then releases it immediately after process_prebuilt has consumed top-k and hidden state into the prebuilt batch. Abort paths also release any held decode metadata slot.
Constraint: Decode disaggregation metadata buffers are reusable slot views consumed later by process_prebuilt.
Rejected: Clone transferred EAGLE tensors at commit time | correct but less efficient and masks the ownership contract.
Rejected: Release in process_batch_result_prebuilt | holds slots across forward longer than needed.
Confidence: medium
Scope-risk: moderate
Directive: Do not free successful EAGLE transfer metadata in pop_transferred unless process_prebuilt consumption is also moved earlier.
Tested: Remote py_compile for decode.py, scheduler.py, scheduler_output_processor_mixin.py, and test_decode_queue_compaction.py.
Tested: Remote focused lifecycle tests passed: 3 passed.
Tested: Remote full test_decode_queue_compaction.py passed: 11 passed, 5 warnings.
Not-tested: Fresh ETE runtime validation of EAGLE accept-length recovery after the C48 sync.
Decode committed EAGLE top-k and hidden-state tensors as views into reusable metadata-buffer rows. The metadata index is freed immediately after transfer commit, while the request may wait before process_prebuilt consumes the draft state. Under concurrent cache-hit traffic a later transfer can overwrite the same row, leaving output_id copied correctly but EAGLE draft state corrupted, which matches low accept length despite successful KV/state registration.
Constraint: Metadata slots are intentionally recycled right after transfer commit for throughput.
Rejected: Hold metadata slots until process_prebuilt | larger lifetime change and reduces transfer capacity; cloning the small prebuilt EAGLE state is narrower.
Confidence: high
Scope-risk: narrow
Directive: Do not store reusable metadata-buffer views on Req unless the slot lifetime is extended through all consumers.
Tested: Local py_compile for decode.py and test_decode_queue_compaction.py.
Tested: Remote g0034 container py_compile for decode.py and test_decode_queue_compaction.py.
Tested: Remote g0034 focused clone-lifetime test: 1 passed.
Tested: Remote g0034 test_decode_queue_compaction.py: 10 passed, 5 warnings.
Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
Cache-hit EAGLE/NextN draft was falling back to full materialization while the target path used page-aligned prefix materialize plus fresh current-suffix splice. That creates a target/draft KV source asymmetry exactly on the high-cache-hit path where decode accept length collapsed. The draft model still does not get next-layer async prefetch; only the same-layer current suffix compose contract is made role-agnostic.
Constraint: EAGLE/NextN has no real next decoder layer, so draft async prefetch remains disabled.
Rejected: Restore draft full-materialize fallback | recreates the observed target/draft cache-hit asymmetry and hides stale-current-suffix bugs.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce draft cache-hit current-reuse fallback without proving the draft persistent pool has fully fresh suffix rows before attention.
Tested: Local py_compile for cp_shared_kv_runtime.py and test_cp_shared_kv_runtime.py.
Tested: Remote g0034 container py_compile for changed runtime/test files.
Tested: Remote g0034 pytest test_cp_shared_kv_runtime.py: 73 passed, 5 warnings, 2 subtests passed.
Tested: Remote g0034 related suite test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py: 128 passed, 5 warnings, 2 subtests passed.
Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
The C32 startup audit was diagnostic-only, but it incorrectly read spec_algorithm from helper queue objects. PrefillBootstrapQueue and DecodePreallocQueue keep scheduler-owned runtime state instead, so the debug path failed before KV manager initialization could complete.\n\nUse scheduler.spec_algorithm for both startup audits and record the ownership rule in the page-aligned cache ledger.\n\nConstraint: Queue helpers do not copy every scheduler field onto self.\nRejected: Disable EAGLE accept diagnostics | would lose the draft-transfer evidence needed for the accept-len investigation.\nConfidence: high\nScope-risk: narrow\nDirective: Disaggregation helper diagnostics should read scheduler-owned runtime state unless the constructor explicitly copied the field.\nTested: Local py_compile for prefill.py and decode.py.\nTested: Local git diff --check.\nTested: Synced prefill.py/decode.py to g0034 and remote container py_compile passed.\nNot-tested: Full disaggregated startup and ETE traffic after this fix; user controls runtime launch and load generation.
CP shared KV and HiCache now keep page-aligned physical ownership while preserving valid-token radix semantics. Repeated tiny EAGLE exact hits free duplicate tail pages instead of leaking one allocator page, owner-lane load-back uses page-vector admission/eviction, and single-DP idle schedulers avoid entering an unnecessary MLP-sync collective.
The commit also records the current page-aligned cache contract and adds gated decode-side EAGLE accept diagnostics so future accept-length collapses can be tied to draft KV/state transfer evidence instead of more prefill cache speculation.
Constraint: CP HiCache allocator ownership is page-granular while radix matching remains valid-token based.
Constraint: New diagnostics must be gated and must not alter normal EAGLE, transfer, or cache behavior.
Rejected: Padding short requests to cp_size or 2*cp_size pages | wastes KV capacity and still hides valid-tail lifecycle bugs.
Rejected: Adding more unconditional collectives to prove CP consistency | hot-path collectives previously caused severe performance risk.
Confidence: medium
Scope-risk: broad
Directive: Do not reintroduce silent fallback for CP shared KV/HiCache paths; warning-level fallback or fail-fast is intentional.
Tested: git diff --check
Tested: local py_compile for all modified Python files
Tested: remote g0034 container py_compile for modified Python/test files
Tested: remote g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/managers/test_scheduler_dp_attn_mixin.py => 114 passed, 5 warnings, 2 subtests passed
Not-tested: full ETE traffic rerun after this commit
Not-tested: CUDA/TAI kernel benchmark coverage for all production shapes
The repeated-request hang reproduced on g0034 after the cached-prefix
request created both MLA and index async prefetchers with
prefix_lens=[40320] and extend_lens=[65]. The existing one-page default
only blocked sub-page suffixes, so a barely-over-one-page suffix still
entered the next-layer collective path before any later forward progress
was logged.
Raise the default async prefetch extend gate to one page per CP lane while
keeping the env override. This only gates async prefetcher object
creation; target partial-current reuse still uses the synchronous
page-slot compose/current-splice path when no prefetcher exists.
Constraint: cp_size=8,page_size=64 repeated prompt had extend_len=65 and hung immediately after has_mla=True has_index=True create_result logs
Rejected: Disable partial-current reuse for short extends | that would lose the cache-hit benefit and regress current/full reuse
Rejected: Disable all async prefetch by default | broader performance impact than the observed tiny-suffix failure
Confidence: medium
Scope-risk: moderate
Directive: Do not lower the default below one page per CP lane without ETE proof that repeated cache-hit tiny suffixes no longer hang
Tested: Remote g0034 container py_compile for touched runtime/prefetch/test files; targeted C22 tests passed 2 tests plus 2 subtests; full test_cp_shared_kv_runtime.py passed 73 tests plus 2 subtests
Not-tested: Full multi-node ETE repeated-request run after this threshold change
CP shared KV HiCache now treats non-page-aligned valid-tail nodes as page-owned when a later request extends beyond them. The prefix probe, match path, insert path, and prepared backup start now agree on flooring the reusable prefix to the previous physical page boundary, so prepared backup metadata cannot start mid-page or fail to attach at insertion.
Duplicate frees under CP HiCache now go through a page-safe free helper. Insert and unfinished duplicate ranges free only fully unprotected pages; no-insert completion still releases the right tail owned only by the finishing request.
Constraint: CP HiCache allocator frees whole physical pages even when called with token-granular locs.
Rejected: Partial-page sharing/refcounting | too complex for the current page-as-minimum-unit contract.
Rejected: Fix only prepare_write_backup_for_req | match_prefix and insert would still expose exact valid-tail hits and desynchronize prepared backup length.
Confidence: medium
Scope-risk: moderate
Directive: Do not expose non-page-aligned CP valid-tail hits to extending requests unless partial-page ownership is explicitly implemented end-to-end.
Tested: remote g0034 py_compile for touched files
Tested: remote g0034 test_cp_hicache_metadata.py 97 passed
Tested: remote g0034 test_cp_shared_kv_runtime.py 73 passed
Not-tested: test_cp_shared_kv_layout.py aborts during installed sgl_kernel architecture-specific op loading before assertions
Async MLA/index prefetch is a scheduling optimization, not the correctness contract for target current reuse. Tiny cache-hit suffixes can skip async prefetcher creation while target partial-current reuse still composes page-slot prefix materialization with current KV rows synchronously. CP HiCache radix/device accounting now treats retained valid-tail pages as physical page spans so allocator state stays consistent when logical cache keys are shorter than the retained page.
Constraint: CP shared KV ownership and HiCache residency are page-granular while request-visible cache lengths remain valid-token lengths.
Constraint: Async prefetch can hang or regress on large-prefix tiny-extend traffic and must not be required for current reuse.
Rejected: Treat missing prefetcher as fail-fast for target partial-current reuse | disabled useful current reuse and broke tiny-prefix/tiny-suffix traffic.
Rejected: Keep async prefetcher object with synchronous consume mode | conflates prefetch object existence with current-layer correctness and hides fallback semantics.
Confidence: medium
Scope-risk: moderate
Directive: Do not make current-only or target partial-current reuse depend on MLA/index prefetcher creation; prefetcher objects mean async next-layer work exists.
Tested: Remote g0034 container py_compile for touched modules.
Tested: Remote g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 73 passed.
Tested: Remote g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py -> 90 passed.
Not-tested: Latest full ETE traffic run with GLM-5.1 CP HiCache after this commit.
Not-tested: CUDA kernel-level performance impact of synchronous no-prefetch partial-current compose.
Backed CP HiCache tail splits were floored during match, but write preparation and radix insertion could still observe the raw sub-page prefix. That could skip one token during pre-forward reservation or call CpHiCacheNodeMetadata.split() with a non-page boundary. The shared helper now floors backed partial-tail prefixes before probe, match, and insert.
Constraint: CP HiCache uses page-granular physical ownership while radix keys may keep valid-tail lengths.
Rejected: Let probe and insert keep token-precise tail prefixes | it reintroduces half-page ownership through a different entry point.
Confidence: high
Scope-risk: moderate
Directive: Any CP radix path that can split or reserve against backed HiCache metadata must use the same page-floor policy.
Tested: Remote red tests first for probe and insert failures in g0034 container.
Tested: Remote py_compile for hiradix_cache.py and test_cp_hicache_metadata.py.
Tested: Remote targeted C7 tests: 4 passed, 3 warnings.
Tested: Remote pytest test_cp_hicache_metadata.py test_cp_hicache_load_back_owner_lanes.py: 94 passed, 5 warnings.
Not-tested: Live ETE traffic with concurrent divergent tail prompts.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP HiCache ownership is page-granular, so a backed radix node must not be split inside a padded physical tail page. When a shorter hit would require an interior tail-page boundary, matching now floors to the previous page boundary and sacrifices the sub-page cache prefix instead of splitting ownership metadata.
Constraint: Host/device/draft CP HiCache metadata tracks page owners and padded physical spans.
Rejected: Split one padded tail page across two radix nodes | it would require half-page ownership semantics and risks double-counting capacity.
Confidence: high
Scope-risk: moderate
Directive: Keep exact valid-tail hits, but floor partial backed-node splits to page boundaries unless metadata gains explicit sub-page ownership.
Tested: Remote py_compile for hiradix_cache.py and test_cp_hicache_metadata.py in g0034 container.
Tested: Remote pytest targeted backed-tail split tests plus exact valid-tail hit test: 3 passed.
Tested: Remote pytest test_cp_hicache_metadata.py test_cp_hicache_load_back_owner_lanes.py: 92 passed, 5 warnings.
Not-tested: Live ETE traffic under divergent short-prefix prompts.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Mixed prefix/current MLA reuse depends on the page-slot prefetch compose path to preserve padded tail semantics. The compact materialize/current merge path can re-expose suffix slack as valid dense rows, so the backend now raises with an explicit fail-fast marker instead of silently falling back.
Constraint: Page-aligned CP shared KV contract requires suffix tail slack to remain invalid.
Rejected: Keep compact merge as a fallback | it has a different dense-row contract and can hide correctness bugs.
Confidence: high
Scope-risk: moderate
Directive: Do not reintroduce merge_materialized_and_current_kv on MLA partial-current reuse without a page-slot correctness proof.
Tested: Remote py_compile for nsa_backend.py and test_nsa_cp_utils.py in g0034 container.
Tested: Remote pytest test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_mla_partial_current_path_fails_fast_instead_of_compact_fallback.
Tested: Remote pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py: 124 passed, 5 warnings.
Not-tested: Live ETE traffic and CUDA kernel execution for this fail-fast path.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Current reuse, TAI materialize, and TAI fused MLA store are optimized CP shared-KV paths. Their fallback helpers already emitted warnings, but the messages were not consistently grep-able with the standard CP shared-KV fallback marker.
This moves the marker into the helper layer so individual fallback call sites do not have to remember to include it, and keeps the existing per-reason rate limiting.
Constraint: Runtime fallback logs must be visible without enabling debug logging.
Rejected: Prefix only selected call sites | helper-level prefixing avoids future unmarked fallback messages.
Confidence: high
Scope-risk: narrow
Directive: New CP shared-KV runtime fallback helpers should use [CP_SHARED_KV_FALLBACK] and a component name.
Tested: Local py_compile for cp_shared_kv_runtime.py and test_cp_shared_kv_runtime.py.
Tested: Local extracted assertLogs check for all three fallback helper prefixes.
Tested: Remote g0034 pytest exact runtime helper tests: 2 passed, 3 warnings.
Not-tested: Full cp_shared_kv_runtime.py suite in this slice.
Co-authored-by: OmX <omx@oh-my-codex.dev>
The compute-owner allocation path is an intended hot path for CP shared KV. Falling back to legacy page allocation changes the behavior profile enough that INFO/no-marker logging is too easy to miss during profiling.
This makes the fallback warning-visible and gives it the standard CP shared-KV fallback marker while preserving the existing behavior of logging every fallback event.
Constraint: Production fallbacks from intended CP shared-KV hot paths must be visible and grep-able.
Rejected: Keep INFO logging | historical silent or low-visibility fallbacks hid inactive optimized paths.
Confidence: medium
Scope-risk: narrow
Directive: Keep fallback logs warning-level unless the path is proven to be expected steady state and separately observable.
Tested: Local py_compile for common.py and test_cp_shared_kv_layout.py.
Tested: Local extracted assertLogs check for _log_cp_shared_kv_alloc_fallback warning prefix and reason formatting.
Tested: Remote g0034 py_compile for common.py and test_cp_shared_kv_layout.py.
Not-tested: Full remote pytest for test_cp_shared_kv_layout.py; current g0034 container aborts while importing installed sgl_kernel common_ops before test execution.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Mooncake is the only disaggregation transfer backend in this branch with CP shared-KV owner filtering plus logical-page-position destination selection. NIXL still slices destination pages by the original chunk slice, so allowing CP shared-KV prefill on NIXL can silently pair filtered prefill pages with the wrong decode pages.
This keeps the supported path narrow while preserving the page-aligned transfer contract: non-page-aligned valid tails transfer their physical tail page, but do not get padded to CP-size pages.
Constraint: CP shared-KV transfer remaps prefill logical pages to per-rank physical pages while decode metadata remains request-position based.
Rejected: Let NIXL continue through the generic slice path | it lacks logical-page-position selection and can silently corrupt CP shared-KV transfers.
Confidence: high
Scope-risk: narrow
Directive: Do not enable CP shared-KV on another PD transfer backend until its sender filters owner pages and selects decode pages by logical request-page position.
Tested: Local py_compile for server_args and touched tests.
Tested: Remote g0034 pytest test_cp_shared_kv_transfer_mapping.py test_req_to_token_pool.py TestHiCacheArgs: 22 passed, 8 subtests passed.
Not-tested: End-to-end PD transfer with a live non-page-aligned prompt.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Load-back owner-lane eviction can need to evict a device-resident node whose radix value is a valid tail shorter than the physical tail page. The eviction planner now pads that value only to the page boundary before deriving owner counts, matching the existing write/load capacity contract without exposing padding to radix or scheduler lengths.
Constraint: CP HiCache capacity remains page-owner based while radix node values remain valid-token based.
Constraint: Avoid collectives; owner-lane capacity must be deterministic from local metadata and logical page ids.
Rejected: Require device-resident victim values to be page-aligned | valid-tail cache nodes are now an intentional supported state.
Rejected: Pad to cp_size pages | this would waste KV and violate the page-boundary-only contract.
Confidence: medium
Scope-risk: narrow
Directive: If split-inside-tail support is added later, preserve page ownership/refcount semantics before sharing one padded physical page across radix nodes.
Tested: local py_compile for hiradix_cache.py and touched CP HiCache tests.
Tested: remote g0034 new C8 exact tests: 3 passed, 3 warnings.
Tested: remote g0034 CP HiCache impacted suites: 146 passed, 5 warnings.
Tested: remote g0034 CP shared KV C1-C5 suite: 122 passed, 5 warnings.
Not-tested: full local pytest, blocked by missing runtime dependencies such as starlette.
Not-tested: CUDA E2E runtime for this commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP HiCache now keeps radix and scheduler-visible lengths as valid tokens while host/device transfers reserve and replay the padded physical page span. Exact valid-tail write, insertion, and match paths no longer fall back to page-flooring; the physical owner-lane contract still uses padded page metadata.
Constraint: Scheduler prefix indices must never include padded tail locs.
Constraint: Host/device transfer and owner-lane admission remain page-based.
Rejected: Pad to cp_size or 2*cp_size pages | wastes KV and recreates short-tail fallback behavior.
Rejected: Expose padded locs through load_cp return | would leak fake tokens into req.prefix_indices.
Confidence: medium
Scope-risk: moderate
Directive: Do not implement split-inside-tail by duplicating page_owners without a page-sharing/refcount design.
Tested: local py_compile for touched CP HiCache/radix/controller files and tests.
Tested: remote g0034 CP HiCache impacted suites: 143 passed, 5 warnings.
Tested: remote g0034 CP shared KV C1-C5 suite: 122 passed, 5 warnings.
Not-tested: full local pytest, blocked by missing runtime dependencies such as orjson/starlette.
Not-tested: CUDA E2E runtime for this commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared KV and HiCache need a stable contract where physical cache coverage is page-aligned, while scheduler/radix-visible hit length remains the valid token length. This records the contract, adds page-aligned extent metadata, keeps owner assignment on actual tail pages instead of short-prefix fallback, and updates partial current reuse tests around tail-page masking.
Constraint: CP owner lanes operate on page units while scheduler and radix hit accounting must remain token-valid.
Rejected: Pad short suffixes to cp_size or 2*cp_size pages | wastes KV capacity and can turn a small tail into a much larger physical span.
Rejected: Silent direct-write or prefetch fallback | production fallback must be warning-visible for diagnosis.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce replicated short-radix fallback without checking docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md.
Tested: local py_compile for touched runtime, utility, owner, and unit-test files.
Tested: remote g0034 container three-file suite: 122 passed, 5 warnings.
Not-tested: full local pytest, blocked by missing runtime dependencies such as orjson.
Not-tested: CUDA E2E runtime for this commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Cache-hit EAGLE/NextN draft extends can enter the draft DeepEP MoE immediately after CP shared-KV attention. The partial current-reuse path is kept for target layers, but draft cache-hit suffixes now use full materialization until draft has an explicit same-layer reuse contract. Next-layer MLA/index prefetch is also gated by the actual model depth, so the single-layer draft model does not enqueue unused next-layer async work.
The temporary stage traces used to isolate the hang are removed. The retained draft current-reuse fallback is a bounded warning because it changes the runtime path intentionally.
Constraint: EAGLE/NextN has one executable draft layer and mirrors target KV state.
Rejected: Keep partial current reuse for draft cache-hit suffixes | reproduced hangs at draft layer0 before DeepEP MoE completion.
Rejected: Keep temporary stage traces | useful for diagnosis but too noisy for normal runs.
Confidence: medium
Scope-risk: moderate
Directive: Do not re-enable draft cache-hit partial current reuse without an explicit draft same-layer reuse contract and ETE validation with CP shared KV + HiCache + EAGLE.
Tested: py_compile on edited Python files; git diff --check; temp trace grep returned no matches.
Not-tested: Local targeted pytest is blocked by missing pybase64 in this environment; full ETE after log cleanup not run.
Prefix materialize prefetch has fixed launch and coordination overhead, so tiny cache-hit prefixes should stay on the simpler synchronous path. The default threshold is now expressed as a cached import-time token threshold while the existing page override remains available for workload-specific tuning.
Both MLA and index prefix prefetchers use the same page-size-aware threshold, so CP shared-KV prefix prefetch is enabled only when the prefix covers at least one page per CP lane and at least the configured token threshold.
Constraint: The default token threshold must be read once to avoid hot-path env lookups
Constraint: Existing page threshold override must continue to disable or force the gate
Rejected: Hard-code 1024 tokens | tuning needs to be possible without code changes
Rejected: Read token threshold on every maybe_create call | unnecessary hot-path env lookup
Confidence: medium
Scope-risk: narrow
Directive: Keep MLA and index prefetch threshold logic shared; do not let their gates diverge
Tested: Not rerun per user request; prior py_compile, diff check, and isolated env-cache check had passed before commit
Not-tested: full pytest in local environment due missing optional dependencies/kernel import constraints
Not-tested: GLM5 E2E throughput after this threshold change
CP shared KV and HiCache now use owner-lane metadata as the
authoritative capacity view for host write admission and GPU load-back
planning. This removes the debug scalar capacity env and keeps CP load-back
from relying on a rank-wide scalar collective when per-owner availability is
already known. The load-back planner also accounts for evicting child leaves
that unlock ancestor device residency, which fixes small lane deficits despite
large aggregate evictable capacity.
The commit also adds gated CPU timing logs for CP shared-KV MLA/index
prefetch and a CUDA microbenchmark for comparing dense all-reduce with
owner-packed all-gather layouts. The timing logs are intentionally behind the
existing MLA prefetch log env and should not be enabled for throughput
measurements.
Constraint: CP shared KV owner lanes require target/draft capacity decisions to preserve page_owners rather than total-token scalars
Constraint: CUDA collective benchmarks must run on target GPU hosts, not locally
Rejected: Keep SGLANG_CP_HICACHE_CAPACITY_DEBUG observer env | owner-lane admission now replaces that scalar debug path
Rejected: Add a silent scalar-allreduce fallback | unexpected owner-lane mismatch should fail fast or log loudly
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce CP capacity collectives on the scheduler hot path without proving the owner-lane metadata is insufficient
Directive: Disable SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH for end-to-end performance runs; it is diagnostic and high-volume
Tested: git diff --check
Tested: python -m py_compile on changed runtime/test/benchmark Python files
Tested: remote pytest -q test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py (81 passed, 5 warnings)
Not-tested: CUDA benchmark benchmark/hicache/bench_cp_shared_kv_prefetch_collective.py
Not-tested: full GLM5 E2E throughput after this commit
CP shared KV cannot treat capacity as a scalar token count: cache-hit load-back and fresh extend allocation both have to preserve the logical page owner pattern or later direct writes, HiCache reload, and prefix materialization can read the wrong lane. This change moves the critical paths to owner-lane plans, makes owner-lane exhaustion recoverable during prefill scheduling, and routes shared-KV prefix prefetch through prefetch-stream-safe KV getters so HiCache layer-load waits do not attach to the forward stream.
Constraint: CP shared KV correctness depends on page owner lane preservation across allocation, backup, load, eviction, and prefix materialization.
Constraint: Avoid adding CP/global collectives for capacity agreement; derive capacity from deterministic local owner-lane state.
Rejected: Keep SGLANG_DISABLE_TAI_OWNER_SELECT fallback | legacy allocation can silently break owner-lane invariants.
Rejected: Scalar total-token eviction for CP HiCache load-back | total capacity can be sufficient while the required owner lane is exhausted.
Confidence: medium
Scope-risk: broad
Directive: Do not reintroduce silent legacy fallback in owner-lane paths; unexpected owner-lane failure must be warning-level fail-closed or recoverable capacity wait.
Tested: Remote g0034 container PYTHONPATH=python python -m pytest test/registered/unit/mem_cache/test_alloc_pages_with_owners.py test/registered/unit/mem_cache/test_cp_shared_kv_layout.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -q -> 95 passed.
Tested: Local py_compile for modified runtime/cache/scheduler modules.
Not-tested: Full CUDA ETE performance trace for cache-hit overlap and MTP accept-rate impact.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Use tai-kernel for direct page_first_direct per-layer H2D load across MHA, MLA, and NSA indexer pools. This keeps SGLang off the sgl-kernel cudaMemcpyBatchAsync path that crashes on CUDA 13 while preserving fail-fast behavior when the required TAI op is unavailable.
Constraint: remote CUDA 13 stack crashes in sgl-kernel PF->LF direct load via cuMemcpyBatchAsync_v2
Rejected: Silent fallback to sgl-kernel or Python loop | fallbacks would hide either a crash-prone ABI path or a large performance regression
Confidence: high
Scope-risk: moderate
Directive: page_first_direct direct load must remain fail-fast if tai_kernel.nsa_prefill.transfer_kv_per_layer_direct_pf_lf is missing
Tested: remote g0034 PYTHONPATH=python pytest -q test/registered/unit/managers/test_hicache_controller_cp.py: 55 passed, 3 warnings
Tested: remote g0034 CUDA smoke for MLATokenToKVPoolHost.load_to_device_per_layer with direct/page_first_direct passed
Not-tested: full SGLang ETE server after the final commit
CP HiCache write-through under shared KV was still using rank-wide collectives to decide host reservation eviction, and per-layer backup registration could be bypassed before the final forward boundary. This moves backup registration to the final run_batch pre-forward boundary, forwards it through SessionAwareCache, exposes fallback paths as explicit warnings, and introduces deterministic owner-lane capacity planning for CP host reservation.
Constraint: CP shared-KV ranks must keep target and draft host reservations owner-lane consistent without adding hot-path collective synchronization
Constraint: Remote CUDA validation must run in the g0034 container, not locally
Rejected: Keep reserve_slots_max all_reduce as the default admission path | observed reserve collectives reaching double-digit and occasional 100ms+ latency
Rejected: Silent post-forward catch-up backup | hides when per-layer forward-overlap backup is not actually active
Confidence: medium
Scope-risk: broad
Directive: Do not reintroduce CP HiCache hot-path collectives without a measured mismatch case and explicit fallback warning
Tested: py_compile for modified Python modules and CP HiCache metadata test file in remote g0034 container
Tested: python3 -m pytest test/registered/unit/mem_cache/test_cp_hicache_metadata.py -q in remote g0034 container (75 passed, 5 warnings)
Tested: git diff --check HEAD~1..HEAD
Not-tested: Local pytest blocked by missing pybase64 in the local environment
Not-tested: Full CP HiCache + MTP E2E after the no-collective reservation change
Co-authored-by: OmX <omx@oh-my-codex.dev>
