The CP in-seq NSA indexer path was constructing a CUDA validity mask around each fp8_mqa_logits call, synchronizing it back with .item(), and then rebuilding small CUDA tensors before topk. The valid rows are always a contiguous prefix because ke_offset is monotonic, so the count can be derived from existing CP metadata and the path can use slices instead of boolean gather/scatter.
RAGGED topk now reuses the zero row-start vector as the per-row offset override for the single-segment case to avoid cumsum/repeat_interleave metadata kernels after MQA.
Constraint: CP shared KV prefill still needs to preserve prev/next causal visibility and topk transform semantics.
Rejected: Add a new Triton range-builder first | the Python-side synchronization and boolean indexing were the larger immediate risk, and kernel work should be profiled after this smaller change.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce GPU mask .item() or CPU-created CUDA tensors in the fp8_mqa_logits hot path without profiler evidence.
Tested: python3 -m py_compile python/sglang/srt/layers/attention/nsa/nsa_indexer.py test/registered/unit/layers/test_nsa_cp_utils.py
Tested: g0034 docker /sgl-workspace/sglang-tai python3 test/registered/unit/layers/test_nsa_cp_utils.py -> Ran 22 tests OK
Not-tested: full GLM5 PD server throughput/profiler run
Phase8 only gains useful overlap when the next-layer MLA prefix prefetch is allowed to run until the next layer actually consumes the prefetched buffer. The old wait-after-attention switch let runtime configuration collapse the optimization back into current-layer tail latency, so the prefetch path now has one wait policy and the documentation records the implemented behavior.
Constraint: Phase8 should keep the production environment surface minimal while preserving the existing enable and debug-log knobs
Rejected: SGLANG_CP_SHARED_KV_MLA_PREFETCH_WAIT_AFTER_ATTENTION | it reintroduced current-layer synchronous waiting and made profiling behavior depend on a nonessential policy knob
Confidence: medium
Scope-risk: narrow
Directive: Do not add another Phase8 wait policy knob without first proving the added policy improves end-to-end prefill latency under CP shared KV
Tested: Python AST parse for touched Python files
Tested: git diff --check
Not-tested: Full pytest and remote server integration were not run in this commit
Shared CP KV materialization remained on the critical path for cached
NSA/MLA prefill batches. This change introduces a one-layer-ahead
prefetcher that materializes the cached prefix for the next layer on a
separate CUDA stream and consumes it when that layer reaches attention.
The prefetch path keeps the existing dense page-table semantics, defers
waiting until the prefetched buffer is actually consumed, and uses the
TAI optimized materialize/remap helpers when enabled before falling back
to the torch implementation.
The implementation is intentionally gated by environment variables and
keeps layer-2-only probe logging for functional confirmation without
making normal profiling noisy.
Constraint: Prefill CP shared KV must preserve existing page-table and dense KV semantics for NSA paged topk attention
Constraint: The production performance path requires SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1 and logging disabled
Rejected: Wait immediately after the producer layer attention | this truncated the overlap window and hid less work
Rejected: Torch-only prefetch materialize | it bypassed the optimized TAI materialize/remap path and could erase the expected win
Confidence: medium
Scope-risk: moderate
Directive: Do not evaluate Phase8 throughput with SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH=1; use it only to confirm create/start/consume_hit behavior
Tested: Local AST parse for modified Python files
Tested: Local git diff --check
Tested: Remote g0034 container AST parse for modified files under /sgl-workspace/sglang-tai
Tested: Remote g0034 container pytest target covering Phase8 log env, TAI range materialize, optimized slot inverse/remap, and existing token TAI path
Not-tested: Full prefill/decode/router throughput after the TAI prefetch-path fix
Shared KV now relies on page-aligned CP metadata and compute-owner page allocation so persistent MLA KV and NSA index shards can be written by the rank that computed them. The compatibility read path keeps the dense full-view contract for existing topk and attention kernels, but removes duplicated prev/next index materialize, adds optional tai materialize integration, and tightens tests/docs around the fallback boundaries.
Constraint: Decode remains non-CP while prefill CP owns the shared-KV changes
Constraint: Existing attention/topk kernels still expect dense full-view KV/index inputs
Rejected: Change attention kernels to read owner-sharded KV directly | larger semantic change reserved for later phases
Rejected: Merge index K/scale storage with MLA KV storage | would couple topk and attention cache lifecycles before materialize overhead is isolated
Confidence: medium
Scope-risk: broad
Directive: Do not remove fallback logging or debug-gated assertions without reproducing long-context chunked/radix-hit paths
Tested: git diff --check --cached
Not-tested: Local pytest/runtime server verification not run in this commit step per current workflow constraints
Phase 5 needs each current KV page to have exactly one CP compute owner before local KV/index direct writes can be safe. This change teaches in-seq NSA prefill CP to produce page-aligned split metadata under shared-KV mode, threads page size into the metadata builders, and fixes local pair splitting so unequal page-aligned zigzag segments do not corrupt topk inputs.
Constraint: Phase 5 direct-write layout requires page ownership to be expressible at page granularity
Constraint: Short page-unit batches remain on the token-balanced fallback to avoid zero-page segment risk
Rejected: Split local q/weights by half | page-aligned zigzag segments can have unequal token counts
Confidence: medium
Scope-risk: moderate
Directive: Do not enable compute-owner direct writes unless nsa_cp_metadata.page_aligned is true and local loc ownership is verified
Tested: python3 -m py_compile python/sglang/srt/layers/attention/nsa/utils.py python/sglang/srt/layers/attention/nsa/nsa_indexer.py python/sglang/srt/models/deepseek_v2.py python/sglang/srt/models/deepseek_nextn.py test/registered/unit/layers/test_nsa_cp_utils.py
Not-tested: Local pytest collection is blocked in this environment by missing pybase64; container/runtime tests were not rerun during this commit step
The shared-KV materialize path was spending time in Python-observed CUDA tensor predicates and dynamic-shape remap helpers. Keep the runtime changes that move the hot paged path to slot-based device remapping, while dropping the NVTX experiment from this commit so profiling annotations do not become part of the runtime surface yet.\n\nThe MLA read path now passes the real page table as the page remap source, which keeps paged topk indices tied to the same logical page-table domain used to build the dense materialized KV view.\n\nConstraint: CP shared KV still needs a dense per-call view before deeper Phase4/Phase5 layout changes remove the materialize cost.\nRejected: Keep NVTX ranges in this commit | user requested reverting NVTX instrumentation before commit\nRejected: Restore compact unique-page remap everywhere | it reintroduces CUDA sync-prone dynamic-shape ops on the hot paged materialize path\nConfidence: medium\nScope-risk: moderate\nDirective: Benchmark slot-remap buffer size against compact unique-page remap before treating this as the final performance path; Phase4/5 should reduce materialize instead of relying on this aggregation path.\nTested: git diff --check on changed files; python -m py_compile on changed runtime/backend/test files; grep confirmed NVTX symbols removed\nNot-tested: pytest blocked locally by missing pybase64 dependency; multi-node PD runtime not rerun
Phase 2 shared KV keeps persistent KV/index sharded across CP ranks but
uses a full-view compatibility layer before NSA topk and MLA attention.
For current-only prefill chunks, the current KV/index tensors have already
been CP all-gathered and reranged before being written to the sharded
persistent pool. This change adds a guarded current-reuse path that remaps
logical current locs to compact tensor rows and skips the shared-KV
materialize path for current-only MLA and NSA indexer reads.
Constraint: Existing NSA/MLA kernels still consume full-view/compact page tables; history and mixed current/history batches must keep the Phase 2 fallback.
Rejected: Make all history attention shard-aware in this patch | that requires global topk merge and distributed sparse attention and belongs to a later phase.
Confidence: medium
Scope-risk: moderate
Reversibility: clean
Directive: Do not remove the Phase 2 fallback until mixed/history shared-KV paths have correctness and performance coverage.
Tested: python -m py_compile on modified Python files
Tested: git diff --check on staged modified files
Not-tested: local pytest collection is blocked by missing pybase64 in this environment.
Not-tested: full long-context chunked prefill/decode performance in this commit step.
Prefill CP previously replicated NSA/MLA persistent KV on every CP rank, so CP8 consumed eight copies of KV memory while exposing only one rank of logical cache capacity. This change splits logical KV locs from per-rank physical storage, shards MLA latent KV and NSA index K/scale by deterministic page ownership, and keeps existing NSA attention kernels working through a full-view runtime materialization layer.
Mooncake PD transfer now sends each prefill CP rank's owned physical pages with explicit logical page positions so non-CP decode can reconstruct full-layout KV. The implementation is guarded by an explicit server flag and startup checks, and the design documentation records the implemented scope, debug environment, and Phase 3 boundary.
Constraint: Phase 2 must preserve existing NSA attention/index kernels via runtime full-view materialization
Constraint: Decode side remains non-CP and receives full KV through Mooncake
Rejected: Shard-aware NSA attention in this change | belongs to Phase 3 because it requires distributed topk/softmax/output contracts
Rejected: Request-contiguous CP ownership | unstable under chunked prefill and tied to attention split mode
Confidence: medium
Scope-risk: broad
Directive: Do not enable round-robin CP shared KV without wiring runtime materialization/PD transfer contracts for that split mode
Directive: Keep SGLANG_DEBUG_CP_SHARED_KV disabled for perf measurements; it intentionally enables CUDA-syncing diagnostics
Tested: Remote py_compile for shared-KV touched Python files in g0034 container
Tested: Remote pytest selected cp_shared/shared_kv/nsa suite: 37 passed, 34 deselected
Not-tested: Full GLM5 multi-node throughput/regression run after final doc update
Not-tested: Phase 3 shard-aware runtime, round-robin CP mode, and non-Mooncake PD backends
Replace full hidden all-gather at prefill tail with per-request last-token
hidden collection, reducing communication from total_tokens x hidden_size
to cp_size x bs x hidden_size for both in-seq-split and round-robin modes.
- nsa/utils.py: add cp_collect_last_token_hidden() with mode-specific
narrow collection helpers that only gather the last token hidden
- deepseek_v2.py: add _should_use_narrow_output_path() gate on
DeepseekV2Model, fallback to full gather for EAGLE/return_logprob/
capture_hidden batches
- logits_processor.py: add _is_compact_hidden_states() to bypass
_get_pruned_states() when hidden is already compact
