Commit Graph
100 Commits
Author SHA1 Message Date
laoyao0822andOmX dfa168abe9 Prevent duplicate CP HiCache write acks from crashing split drain
CP per-layer write-through can leave duplicated ready ack ids in the
ack queue while pending-split insertion drains write visibility. The
pre-scan observed both duplicates as registered before the first
completion removed radix state, so the second completion raised KeyError
and killed the scheduler.

The write-check path now records completed ack ids until matching queue
entries are drained. Duplicate completed acks are removed with a warning,
while genuinely unknown or unregistered acks still fail fast.

Constraint: CP HiCache pending-split drain may call writing_check outside the normal idle event path.
Rejected: Blind pop(..., None) for all unknown acks | would silently hide truly unattached or corrupt ack state.
Confidence: high
Scope-risk: moderate
Directive: Do not remove the completed-ack short-term memory unless duplicate and stale ack queues are proven impossible across TP MIN synchronization.
Tested: Local py_compile for hiradix_cache.py and test_cp_hicache_metadata.py.
Tested: Remote cjy-glm5-new py_compile and full test_cp_hicache_metadata.py, 121 passed.
Not-tested: Full ETE prefill workload after restarting service.
Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 17:46:22 +08:00
laoyao0822andOmX 9c8e3e99cb Align OpenAI serving behavior with Para deployments
Absorb PR 11's final Para compatibility surface as an opt-in OpenAI serving layer rather than hard-coding business defaults into protocol models. The change adds server args for Para chat defaults, Kimi/GLM compatibility, tool-choice normalization, tool-role text flattening, and streaming first-chunk error preflight while preserving default upstream behavior unless explicitly enabled.

Reasoning token usage is also propagated through chat/completion usage paths, with GLM compatibility emitting completion_tokens_details.reasoning_tokens. Low-risk protocol fixes accept string image_url content parts and preserve GLM function-call argument value whitespace.

Constraint: Online Para-compatible deployments require request/response semantics that differ from default OpenAI serving behavior.

Constraint: Current CP/HiCache/bs>1 work must not be coupled to OpenAI serving compatibility changes.

Rejected: Merge PR 11 history directly | intermediate commits briefly hard-code chat max_tokens=32768 before later gating it by server args.

Rejected: Enable Para compatibility by default | would change non-Para OpenAI-compatible deployments.

Confidence: high

Scope-risk: moderate

Directive: Keep Para-specific serving policies behind explicit server args unless the business contract changes globally.

Tested: PYTHONPATH=python:. python -m unittest discover -s test/registered/unit/entrypoints/openai -p 'test_para_serving_protocol.py' -v (19 tests OK)

Tested: python -m py_compile modified OpenAI serving, tokenizer manager, server_args, function-call detector, and test files

Not-tested: Live router/prefill/decode OpenAI serving E2E after enabling Para flags.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 05:59:08 +08:00
laoyao0822andOmX 8cfdb0466e Reduce CP per-layer transfer success-log noise
Per-layer prefill-to-decode transfer now finishes per request/rank/chunk, so success-path INFO logs can dominate production logs and hide actual failures. Keep successful finish breakdown and completion messages at DEBUG while preserving nonzero finish status as WARNING.

Constraint: Per-layer transfer is a hot path under CP shared-KV and may produce many batch completions per request.

Rejected: Disable CP per-layer transfer logging entirely | failures still need visible warning-level evidence.

Confidence: high

Scope-risk: narrow

Directive: Do not promote successful per-request transfer completion logs back to INFO without rate limiting.

Tested: PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_cp_per_layer_transfer.py::TestPerLayerTransferContext::test_successful_finish_does_not_emit_hot_path_info_log

Tested: python -m py_compile python/sglang/srt/disaggregation/cp_per_layer_transfer.py python/sglang/srt/disaggregation/mooncake/conn.py

Not-tested: Full local disaggregation suite blocked by missing local orjson dependency.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 05:49:05 +08:00
laoyao0822andOmX 7284a469a2 Reuse prepared HiCache load descriptors across CP prefill layers
CP shared-KV bs>1 cache-hit loads already merge request load ops, but the host pool still rebuilt layer-invariant mapping work from the same host/device indices. Introduce a PreparedLoadDescriptor lifecycle around begin/end load, wire MLA KV and NSA index H2D loads through tai-kernel prepared submit when available, and add timing hooks plus regression coverage for descriptor reuse and explicit fallback logging. Record the P4/P6b design and benchmark results in the advanced feature notes.

Constraint: Radix residency and allocator decisions remain synchronous; only the data-transfer descriptor is prepared for per-layer async submit.

Constraint: Production fast path must not silently fall back when tai prepared H2D support is missing.

Rejected: Cross-batch descriptor reuse | descriptor lifetime and tensor ownership are only safe within one load operation.

Rejected: Change L2->L1 scheduling to layer-ahead prefetch in this commit | that is a separate lifecycle change after descriptor reuse is stable.

Confidence: medium

Scope-risk: moderate

Directive: Keep LayerDoneCounter per-layer readiness semantics; do not replace with all-layer waits.

Tested: python -m py_compile python/sglang/srt/mem_cache/memory_pool_host.py python/sglang/srt/managers/cache_controller.py

Tested: Remote g0034:cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py (88 passed)

Tested: Remote tai-kernel prepared descriptor CUDA test (6 passed) and P4 benchmark full matrix (90 rows)

Not-tested: ETE replay/GSM8K cache-hit correctness after this commit

Not-tested: Layer-ahead L2->L1 prefetch scheduling

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 05:09:41 +08:00
laoyao0822andOmX adf357b02c Default CP bs>1 extend admission to chunk budget
When chunked prefill is active, CP shared-KV bs>1 cannot consume more extend
tokens than the current chunk budget. If the CP-specific extend-token limit is
omitted, default it to rem_chunk_tokens so scheduler admission reflects the
reachable chunk capacity. The request-count and cached-token knobs keep their
None-as-unlimited behavior.

Constraint: CP bs>1 batching must not advertise a larger extend batch than chunked prefill can execute.
Rejected: Require users to always set --cp-shared-kv-prefill-max-total-extend-tokens | the safe default is already available from chunked prefill state.
Rejected: Default batch request or cached-token limits | those are policy knobs and None should remain unlimited.
Confidence: high
Scope-risk: narrow
Directive: Keep --cp-shared-kv-prefill-max-total-extend-tokens as min(user_limit, chunk_budget) when both exist.
Tested: Local py_compile for schedule_policy.py and test_prefill_adder.py.
Tested: Remote g0034 cjy-glm5-new targeted prefill_adder tests: 2 passed.
Not-tested: Full ETE scheduler batching distribution after defaulting the extend limit.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 03:51:41 +08:00
laoyao0822andOmX d696039092 Align MQA logits admission with CP split runtime shape
CP shared-KV batching previously estimated MQA logits from full request
extend/context rows, which overstated memory because CP in-seq split only
computes each rank's two zigzag segments. Add CP-size aware row accounting
that mirrors the fused CP MQA materialization path and take the worst local
rank peak for scheduler admission.

Expose SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_GB as a more direct cap for one fp32
MQA logits chunk. Runtime and scheduler now both translate this GB cap into
chunk rows from the actual K rows, while keeping the old row cap as a
mutually-exclusive expert override.

Constraint: Scheduler admission must stay CUDA-sync-free and use static budget information only.
Rejected: Keep full-request q*k admission | it over-gates CP bs>1 batches because CP splits q rows per rank.
Rejected: Let rows and GB caps both apply | precedence would be ambiguous during tuning.
Confidence: medium
Scope-risk: moderate
Directive: Keep MQA logits admission tied to the fused CP MQA segment shape; do not revert to full request token counts.
Tested: Local py_compile for touched runtime, scheduler, estimator, and tests.
Tested: Local pytest test_cp_shared_kv_prefill_buffer_estimator.py: 8 passed.
Tested: Remote g0034 cjy-glm5-new py_compile and targeted estimator/runtime tests: 13 passed.
Not-tested: Full ETE high-cache-hit CP bs>1 load with SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_GB.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 03:51:20 +08:00
laoyao0822andOmX 250fab291d Account for MQA logits in CP batch admission
CP shared-KV bs>1 admission already bounds request count, extend tokens,
cached tokens, and an estimated temporary buffer size. The estimate missed
the fp32 MQA logits temporary, whose peak grows with query rows times
context rows and can dominate high-cache-hit multi-request batches.

Add an MQA logits peak term to the CPU-only estimator and include it in
the layer-forward peak enforced by --cp-shared-kv-prefill-max-buffer-size.
When SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_ROWS is set, admission estimates the
post-chunk peak using that row cap; otherwise it remains conservative and
assumes the full extend-row count.

Constraint: Scheduler admission must stay CPU-only and cannot query CUDA free memory.
Rejected: Add a separate scheduler limit for MQA logits | the existing max-buffer-size knob is the right aggregate admission budget.
Rejected: Use SGLANG_NSA_MQA_LOGITS_FREE_MEM_FRACTION in scheduler | that depends on runtime CUDA free memory and would make admission host-sync or stale.
Confidence: medium
Scope-risk: moderate
Directive: Keep the estimator conservative when chunk max rows is unset; do not rely on CUDA free-memory queries in scheduler admission.
Tested: Local py_compile for estimator, scheduler, schedule_policy, and estimator tests.
Tested: Local pytest test_cp_shared_kv_prefill_buffer_estimator.py: 5 passed.
Tested: Remote g0034 cjy-glm5-new py_compile and estimator pytest: 5 passed.
Not-tested: ETE scheduler admission under high-cache-hit bs>1 traffic.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 03:25:22 +08:00
laoyao0822andOmX ddc1233955 Bound CP MQA logits buffers with row chunking
CP shared-KV bs>1 can build large fp32 MQA-logits temporaries from
DeepGEMM fp8_mqa_logits. The official SGLang path already chunks normal NSA
MQA logits by query rows behind a cached memory budget; carry the same budget
control into our NSA indexer and extend it to CP-ragged topk paths that use
row-wise topk_indices_offset_override.

This keeps the previous one-time cached memory-budget behavior rather than the
recent current-free-mem per-forward variant that regressed performance. A new
optional max-rows env provides an explicit hard cap for debugging or controlled
ETE runs without adding host syncs.

Constraint: DeepGEMM materializes fp32 [q, k] logits internally, so row chunking is the narrowest way to cap temporary memory
Rejected: Restore the reverted syh current-free-mem implementation | it changed hot-path heuristics and showed poor runtime performance
Rejected: Split by K/context dimension | would change topk semantics and require a different transform contract
Confidence: medium
Scope-risk: moderate
Directive: CP-ragged chunking relies on topk_indices_offset_override being row-addressed; do not route non-ragged CP paths through it without separate validation
Tested: Local py_compile for environ.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py
Tested: Remote g0034 cjy-glm5-new py_compile for environ.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py
Tested: Remote pytest TestCpSharedKVTaiMaterializeIntegration, 17 passed
Not-tested: CUDA ETE high-cache-hit bs>1 workload memory/performance after chunking
Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 03:15:52 +08:00
laoyao0822andOmX e0ea8a485c Revert NSA MQA logits chunking while performance is unresolved
Temporarily revert the syh MQA-logits row-chunking port because the observed
runtime performance is worse on the current bs>1 CP prefill workload. Keep the
history explicit so the optimization can be revisited after profiling identifies
where the extra overhead comes from.

This reverts commit 4e49751406 (Bound NSA MQA logits peak memory).

Constraint: Current priority is restoring the faster known path for remote ETE runs
Rejected: Keep chunking behind the force/debug env only | the production auto-chunk path still changes runtime heuristics and should not stay until profiled
Confidence: high
Scope-risk: narrow
Directive: Reintroduce logits chunking only with ETE performance evidence and forced-chunk equivalence coverage
Tested: Local py_compile for environ.py and nsa_indexer.py
Not-tested: Remote ETE performance after revert
Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 03:04:03 +08:00
laoyao0822andOmX 4e49751406 Bound NSA MQA logits peak memory
Paged and CP-ragged NSA indexer paths could materialize q x context fp32
MQA-logits buffers large enough to OOM high-cache-hit bs>1 prefill batches.
Port the syh branch chunking logic so paged and ragged paths split logits by
query rows when the estimated logits buffer exceeds the current free-memory
budget.

The free-memory query is cached on forward_batch so the OOM guard uses current
free memory without adding a torch.cuda.mem_get_info host sync on every layer.
The only new env kept from the syh commits is
SGLANG_NSA_MQA_LOGITS_CHUNK_FORCE_ROWS, which forces chunking for equivalence
validation.

Constraint: DeepGEMM fp8_mqa_logits still materializes fp32 logits internally, so limiting q rows is the least invasive way to cap peak memory
Rejected: Carry unrelated syh envs for page trace/source-fingerprint strictness | not part of the logits peak-memory fix
Rejected: Static mem_fraction-only budget | overestimates logits headroom shared with other forward activations
Confidence: medium
Scope-risk: moderate
Directive: Keep chunking row-split only; changing K/context partitioning needs topk_transform equivalence validation
Related: 40a0389a9c feat(nsa): chunk paged + CP-ragged MQA-logits by current-free-mem budget
Related: 108fa1f538 perf(nsa): cache MQA-logits free-mem budget per-forward
Tested: Local py_compile for environ.py and nsa_indexer.py
Tested: Remote g0034 cjy-glm5-new py_compile for environ.py and nsa_indexer.py
Not-tested: CUDA ETE run with forced SGLANG_NSA_MQA_LOGITS_CHUNK_FORCE_ROWS equivalence check
Not-tested: Full high-cache-hit bs>1 prefill OOM regression workload
Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 03:03:01 +08:00
laoyao0822 cc908dd556 Reuse DeepGEMM JIT cache across restarts
New sgl-deep-gemm wheels persist cubins, but SGLang was still replaying the full warmup envelope after every restart. The wrapper also translated SGLANG_DG_* env after importing deep_gemm, so the wheel could observe stale cache settings.\n\nMove DeepGEMM JIT env preparation before the first import, bridge the current SGLANG_DG_USE_NVRTC spelling, replace dense M walks with category-specific sparse M grids, and add a SGLang-side warmup manifest under the DeepGEMM cache dir. A manifest hit skips only the SGLang replay loop; missing or stale cache entries still force warmup and refresh the manifest.\n\nConstraint: sgl-deep-gemm 0.1.2 removed the compile-mode API, so dense 1..m_max loops launch real kernels.\nConstraint: DeepGEMM cache keys include compiler flags such as the deep_gemm include path; path-stable environments are still required for cross-container cache hits.\nRejected: SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 | bypasses warmup instead of making cache reuse correct.\nRejected: Enable per-process cache dirs by default | isolates corruption but defeats persistent cross-process cache reuse.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce dense DeepGEMM warmup for new wheels unless compile-mode support is verified on the target wheel.\nTested: Local py_compile for jit_cache.py configurer.py compile_utils.py model_runner.py\nTested: Local pytest test/registered/unit/layers/test_deep_gemm_jit_cache.py (6 passed)\nTested: Remote g0034 cjy-glm5-new py_compile for same files\nTested: Remote g0034 cjy-glm5-new pytest test/registered/unit/layers/test_deep_gemm_jit_cache.py (6 passed)\nTested: Remote env probe verified SGLANG_DG_CACHE_DIR and SGLANG_DG_USE_NVRTC bridge to DG_JIT_* before configurer completes\nNot-tested: Full decode cold-cache/hot-cache restart timing and manifest-hit ETE log validation
2026-06-11 02:55:04 +08:00
laoyao0822 3a43727216 Bound CP prefill batching by estimated temp memory
CP shared-KV bs>1 batching was only bounded by request count, extend tokens, and cached tokens. That left temporary GPU buffers such as MLA/index materialization, remap metadata, logits windows, and transfer descriptors implicit, and raw extend-token limits could exceed the active chunked-prefill budget.\n\nThis adds an explicit max-buffer-size admission gate with a CPU-only stream-aware estimator, wires it through PrefillAdder/Scheduler, performs a startup CUDA smoke allocation when configured, and reports the estimate in the scheduler admission benchmark. When chunked prefill is active, the effective CP extend-token limit is capped by the current chunk budget so the CP path does not advertise unreachable batch capacity or lift max-prefill-tokens too far.\n\nConstraint: Admission estimation must stay CPU-only on the scheduler hot path; CUDA allocation is limited to startup smoke checking.\nConstraint: Single oversized requests must still be allowed to run alone to avoid scheduler deadlock.\nRejected: Rely only on --max-prefill-tokens | it does not reliably bound the first oversized request and does not model cache-hit/load-back pressure.\nRejected: Let CP extend limit exceed chunked-prefill size | it creates an unreachable effective capacity and misleading budget lift.\nConfidence: medium\nScope-risk: moderate\nDirective: If bs>1 L1 prefetch is enabled later, update CPSharedKVPrefillBufferEstimatorContext.bs_gt1_l1_prefetch_enabled and include the live prefetch dense buffers in overlap windows.\nTested: local py_compile for touched files\nTested: local PYTHONPATH=python pytest -q test/registered/unit/managers/test_cp_shared_kv_prefill_buffer_estimator.py (4 passed)\nTested: remote cjy-glm5-new targeted pytest for new server_args, PrefillAdder, estimator, and benchmark cases (10 passed)\nTested: remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/managers/test_cp_shared_kv_prefill_buffer_estimator.py test/registered/unit/managers/test_prefill_adder.py test/registered/unit/managers/test_prefill_scheduler_admission_bench.py (29 passed before chunk cap, then test_prefill_adder.py 21 passed after chunk cap)\nNot-tested: full server_args suite because existing TestPrepareServerArgs tries to reach HuggingFace and fails under container DNS/network\nNot-tested: GLM5 ETE smoke with --cp-shared-kv-prefill-max-buffer-size
2026-06-11 01:33:28 +08:00
laoyao0822 9a9893e571 Model CP scheduler admission with cache-hit pressure
Add an offline benchmark that reuses PrefillAdder to model how L1 cached tokens, L2 HiCache hits, and extend tokens shape CP shared-KV prefill batch admission. The tool makes scheduler stop reasons and fake L2 load-back capacity pressure observable without starting a model.

Constraint: The benchmark must stay CPU/offline and avoid depending on CUDA execution or live services.

Constraint: L2 cached tokens are modeled as host_hit_length, so successful load-back both increases prefix_len and consumes fake L1 capacity.

Rejected: Build an ETE benchmark first | too slow for isolating scheduler admission behavior.

Rejected: Reimplement scheduler logic from scratch | would drift from PrefillAdder semantics.

Confidence: high

Scope-risk: narrow

Directive: Treat duration_us as Python admission overhead only; it is not an ETE latency metric.

Tested: Remote pytest test/registered/unit/managers/test_prefill_scheduler_admission_bench.py: 4 passed as part of 6 targeted tests.

Tested: Remote synthetic benchmark run with --cp-max-total-cached-tokens showed second 4096-token cached request stopped with OTHER.

Not-tested: Real traffic trace import from production logs.
2026-06-10 22:21:59 +08:00
laoyao0822 4f65d7a176 Bound CP prefill batches by cached-token pressure
High cache-hit CP shared-KV batches can have small extend tokens while still carrying substantial prefix/load-back work. Add a CP-specific cached-token admission limit so operators can bound that pressure independently from extend-token batching.

Constraint: The limit must not deadlock a single high-cache-hit request; it only stops adding additional requests to a non-empty batch.

Constraint: Cached tokens are counted after L2 load-back planning via prefix_len, so L1 hits and successful L2 hits share one scheduler budget.

Rejected: Reuse max_prefill_tokens | it limits generic input budget and does not represent cached-token work.

Rejected: Count only L1 prefix before load-back | would miss L2 hit pressure, which is one of the target cases.

Confidence: high

Scope-risk: moderate

Directive: Keep cached-token and extend-token limits separate; they bound different scheduler costs.

Tested: Remote pytest targeted cached-token PrefillAdder cases: 2 passed.

Tested: Remote pytest test/registered/unit/managers/test_prefill_adder.py: 18 passed.

Tested: Remote ServerArgs CP validation smoke: SERVER_ARGS_CACHED_LIMIT_OK.

Not-tested: Full ETE replay with a production cached-token limit value.
2026-06-10 22:21:46 +08:00
laoyao0822 342c552ab3 Decouple CP prefill batching from generic token cap
CP shared-KV bs>1 uses cp_shared_kv_prefill_max_total_extend_tokens as its grouping admission limit, but the generic max_prefill_tokens budget could still stop batching earlier. Raise only the legacy input-token admission budget for this CP path while keeping allocator-owned capacity checks unchanged.

Constraint: CP shared-KV bs>1 needs large cache-hit batches without relying on the generic max_prefill_tokens default.

Constraint: Allocator capacity must remain enforced by rem_total_tokens, cur_rem_tokens, and prepare_for_extend().

Rejected: Increase server-wide max_prefill_tokens | would change generic scheduler behavior and non-CP paths.

Confidence: high

Scope-risk: narrow

Directive: Do not use max_prefill_tokens as the CP shared-KV bs>1 grouping limit; use the CP-specific total extend token knob.

Tested: Local py_compile for schedule_policy.py and test_prefill_adder.py.

Tested: Remote pytest targeted CP PrefillAdder cases: 5 passed.

Tested: Remote pytest test/registered/unit/managers/test_prefill_adder.py: 16 passed.

Not-tested: Full ETE replay after this scheduler-only change.
2026-06-10 21:36:22 +08:00
laoyao0822 ffff715f00 Plan bs>1 CP shared-KV L1 prefetch safely
Record the staged design for batch-aware L1 shared-KV prefetch under CP + HiCache. The plan separates L2->L1 load two-layer lookahead from L1 prefetch one-layer lookahead, keeps target first and draft/EAGLE out of the first phase, and calls out that current suffix reuse remains synchronous until the request-aware span contract is extended.

Constraint: Existing CP shared-KV correctness failures came from batch-global remap/req-id ambiguity, so bs>1 prefetch must carry explicit per-request spans and ids.

Rejected: Reuse the bs=1 scalar prefix_pages prefetcher for bs>1 | it aliases request rows and cannot represent mixed prefix/current spans.

Rejected: Make draft/EAGLE prefetch part of the first phase | target path must be validated first before coupling draft lifecycle.

Confidence: medium

Scope-risk: narrow

Directive: Do not open bs>1 L1 prefetch until req-id fail-fast and per-request slot span tests cover both MLA and index paths.

Tested: git diff --cached --check

Not-tested: Runtime implementation; this is a design document only.
2026-06-10 07:04:39 +08:00
laoyao0822 f78c414b64 Skip unused FP8 NSA K-cache dequant work
High cache-hit NSA sparse attention only gathers rows selected by topk_indices, but the FP8 RAGGED path dequantized the whole materialized K buffer first. Wire the syh in-place topk-only dequant path behind an explicit env gate so the sparse path can dequant referenced rows at their original row ids while keeping topk_indices unchanged. The old full-dequant path remains the default and the verification gate stays available for small correctness checks only.

Constraint: Production runs should enable only SGLANG_NSA_DEQUANT_ONLY_TOPK=1; VERIFY also runs the full-dequant reference and is too expensive for perf tests.

Rejected: Compact/remap topk buffer | data-dependent shape and remap add synchronization/aliasing risk; syh final in-place contract avoids both.

Confidence: medium

Scope-risk: moderate

Directive: Do not enable SGLANG_NSA_DEQUANT_ONLY_TOPK_VERIFY in throughput runs; use it only for small correctness probes.

Tested: python -m py_compile python/sglang/srt/environ.py python/sglang/srt/layers/attention/nsa_backend.py test/registered/unit/layers/test_nsa_dequant_only_topk.py

Tested: git diff --check

Tested: remote g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_dequant_only_topk.py -> 2 passed

Tested: remote g0034 CUDA smoke for tai_kernel.nsa_prefill.nsa_dequant_topk_inplace topk rows matched torch reference

Not-tested: Full GSM8K or replay ETE with SGLANG_NSA_DEQUANT_ONLY_TOPK=1
2026-06-10 07:00:43 +08:00
laoyao0822 c8c736e75c new pyprojectoml 2026-06-10 05:54:43 +08:00
laoyao0822andOmX af20334f4c Size CP HiCache host budget from compact index layers
The CP shared target+draft HiCache budget was still estimating NSA target
bytes per token as if every target layer had an index cache. After index
skip compacts L1 and L2 index buffers to active layers, that stale estimate
kept host/L2 token capacity artificially low even though the physical host
index allocation was already smaller.

Use the pool's index_active_layer_ids when estimating NSA HiCache bytes per
token, with the existing full-layer fallback for pools that do not expose
compact metadata. This keeps the shared target+draft capacity computation
consistent with the actual L2 host layout.

Constraint: Index skip compacts NSA index cache by active logical layers, but MLA KV remains full-layer.
Rejected: Keep full-layer HiCache estimate | preserves correctness but wastes the L2 capacity recovered by compact index allocation.
Confidence: high
Scope-risk: narrow
Directive: Keep HiCache byte estimates aligned with NSATokenToKVPool allocation metadata; otherwise target+draft token budgets will drift from real host memory use.
Tested: Remote container targeted pytest for active-index HiCache estimate and shared-budget tests -> 3 passed.
Tested: Remote container P3-P6 regression suite plus new capacity tests -> 203 passed, 2 subtests passed.
Not-tested: Full ETE replay/GSM8K after this budget correction.
Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-10 05:53:05 +08:00
laoyao0822andOmX 1ebde44e59 Compact skipped NSA index-cache state safely
Index skip reduces the number of target layers that own NSA index state,
but PD transfer and HiCache still assumed dense full-layer state buffers.
This change carries explicit state layer IDs through prefill/decode
registration, compacts device and host index buffers to active layers,
and maps logical layer IDs to compact slots on transfer paths.

The PD side fails fast when prefill/decode disagree on NSA state layer
identity instead of silently truncating or copying mismatched buffers.
Host direct tests now use the same CPU-index descriptor contract required
by the TAI cudaMemcpyBatchAsync path, and host registered memory is
unregistered on tensor finalization to avoid stale cudaHostRegister state
across CUDA tests.

Constraint: CP shared-KV with index_topk skip must keep target/draft state identity explicit before compacting buffers
Constraint: Direct HiCache TAI transfer rejects CUDA indices to avoid hidden D2H copies on the control path
Rejected: Keep full-layer L1/L2 index buffers | wastes the memory/bandwidth that index skip is meant to save
Rejected: Infer state buffer order by count only | can silently corrupt cache when active layer sets differ
Confidence: high
Scope-risk: moderate
Directive: Do not compact or reorder NSA state buffers without carrying logical layer IDs through PD registration and validating both sides
Tested: Remote container py_compile for touched runtime files
Tested: Remote container pytest: test_nsa_pool_host_unit.py, test_model_runner_kv_cache_mixin.py, test_cp_shared_kv_transfer_mapping.py, test_pd_state_layer_ids.py, test_cp_per_layer_transfer.py, test_cp_shared_kv_runtime.py -> 200 passed, 2 subtests passed
Not-tested: Full ETE GSM8K/replay after compacted P3-P6 changes
Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-10 05:37:06 +08:00
laoyao0822 d21952b903 Reduce inactive NSA index-cache transfer safely
Centralize the IndexCache skip formula and thread the resulting active logical index layers into NSA KV pools. HiCache now skips only the indexer H2D/D2H payload for inactive target layers while preserving per-layer MLA KV transfer, keeping allocation shape unchanged for this phase.

Constraint: P0-P2 must not compact device or host allocation yet; prefill/decode state transfer still has no logical layer-id metadata.

Rejected: Recompute the skip formula separately in mem_cache | formula drift would corrupt cache or waste transfers when offset/pattern settings change.

Rejected: Skip whole-layer HiCache load/backup | MLA KV remains required for every attention layer.

Confidence: medium

Scope-risk: moderate

Directive: Before enabling compact state buffers or compact allocation, add layer-id metadata validation to PD transfer.

Tested: Local py_compile for touched files; remote pytest in g0034 container: test_nsa_index_layers.py and TestNSAIndexerPageIndices, 20 passed.

Not-tested: ETE replay/GSM8K with --nsa-index-topk-freq 4; PD state-transfer compaction remains unimplemented.
2026-06-10 04:28:26 +08:00
laoyao0822 6229c7da60 Plan index-skip cache and transfer reduction
Record the phased design for turning NSA index-topk sharing from a compute-only optimization into HiCache and PD-transfer savings. The plan deliberately separates metadata validation from allocation compaction so prefill/decode state-slot mismatches fail before any bandwidth reduction can corrupt cache contents.

Constraint: CP shared-KV cache corruption previously appeared only on cache-hit paths, so state-layer mapping must be explicit before compact transfer or allocation.

Rejected: Compact L1/L2 index buffers first | without state_layer_ids this can silently map different logical layers to the same transfer slot across prefill and decode.

Confidence: high

Scope-risk: narrow

Directive: Do not skip the state_layer_ids phase before compacting NSA index state buffers.

Tested: Documentation placeholder scan and referenced source-path existence check.

Not-tested: Runtime behavior; this commit is documentation only.
2026-06-10 04:28:26 +08:00
laoyao0822 a32050d1d8 Expose NSA index sharing as launch-time overrides
Index-topk sharing previously required editing model config or passing raw JSON overrides. Add first-class server args that merge into json_model_override_args before ModelConfig is read, preserving existing JSON overrides while letting launch scripts toggle index_topk_freq directly. Treat offset 0 as unset so wrappers can use 0 for default behavior without injecting a model-config override.

Constraint: Prefill and decode launch commands need the same effective model config without mutating /ssd model files.

Rejected: Require editing config.json | operationally fragile across g0034/g0035/g0036 model copies.

Rejected: Only use --json-model-override-args | works but is too error-prone for frequent launch-command tuning.

Confidence: high

Scope-risk: narrow

Directive: Keep these flags as model-config override shortcuts; apply them before any get_model_config() call.

Tested: Remote pytest in g0034 container for NSA index override parser tests: 2 passed.

Tested: py_compile for server_args.py and test_server_args.py.

Not-tested: Full prefill/decode ETE launch with --nsa-index-topk-freq enabled.
2026-06-10 03:24:33 +08:00
laoyao0822 24da983ff5 Enable CP HiCache direct transfers to use layer-page host layout
CP shared-KV HiCache transfers are per-layer, so the host layout should match the access pattern instead of forcing page-major strides through every layer. This adds a direct-only layer_page_first layout, routes per-layer KV and NSA index backup/load through the TAI LF<->LPF direct kernels, and keeps storage/page-buffer metadata paths fail-fast until their page-level contract is redesigned.\n\nThe direct controller keeps host indices in caller order for both page_first_direct and layer_page_first because the TAI direct path requires CPU index descriptors and owns descriptor coalescing. All-layer backup intentionally loops over per-layer direct kernels rather than using the sgl-kernel all-layer direct ABI.\n\nConstraint: layer_page_first is currently host-only CP HiCache; storage backends assume page-major contiguous page metadata.\nConstraint: TAI LPF direct kernels require CPU int64 page indices and complete page spans.\nRejected: silently fallback to SM copy when TAI LPF kernels are missing | that hides production performance regressions.\nRejected: support storage page metadata in this commit | LPF requires a layer-page-level storage contract, not a one-pointer-per-page contract.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not enable storage or kernel backend for layer_page_first without redesigning page-buffer metadata and adding remote ETE coverage.\nTested: local py_compile for touched runtime files.\nTested: remote py_compile in g0034 container for touched runtime files.\nTested: remote targeted pytest: 5 passed for parser/storage/layout/move_indices smoke coverage.\nNot-tested: full CP HiCache ETE with --hicache-mem-layout layer_page_first after this commit step.\nNot-tested: combined CUDA roundtrip tests in one pytest process; previous independent runs passed but combined run exposed a host-memory registration lifecycle issue.
2026-06-10 02:03:07 +08:00
laoyao0822 5e22279670 Carry CP-local hidden contract into EAGLE v2 prefill draft extend
SPEC_V2 builds an EAGLE draft input during target prefill. Under CP shared-KV, the target model may expose draft_hidden_states as a CP-local side channel before CP output collection. The v2 path was still passing global hidden_states and never marked the draft input as CP-local, unlike the non-v2 path.

Thread cp_local_hidden_states through the v2 _draft_extend_for_prefill helper and prefer draft_hidden_states when present. This preserves the semantic marker consumed by the static MLP-sync padding guard without changing the non-CP path.

Constraint: Absorb syh 5562937cf only; SPEC_V2 remains opt-in and broader SPEC_V2-on-CP validation is still separate.
Rejected: Infer CP-local hidden from tensor length | tensor length is ambiguous under static padding and bs>1 compute padding.
Confidence: high
Scope-risk: narrow
Directive: Keep EagleDraftInput.cp_local_hidden_states as an explicit semantic contract; do not replace it with shape-based inference.
Tested: Remote g0034 container red test failed before implementation with unexpected cp_local_hidden_states kwarg.
Tested: Remote g0034 container py_compile for eagle_worker_v2.py and test_nsa_cp_utils.py.
Tested: Remote g0034 container pytest target EAGLE marker tests: 2 passed.
Tested: Remote g0034 container pytest test_nsa_cp_utils.py -k eagle: 2 passed, 99 deselected.
Not-tested: Full ETE with SGLANG_ENABLE_SPEC_V2=1 on CP prefill.
2026-06-09 22:42:54 +08:00
laoyao0822 df2a3696cd Gate CP split validation to eligible extend forwards
CP shared-KV marks all forwards with uses_cp_shared_kv, but TARGET_VERIFY/decode style forwards may legitimately carry no extend prefix page-plan. The CP split helper previously ran the hard page-plan validator before checking context-parallel extend eligibility, so non-extend spec paths could fail before the later no-split decision.

Hoist the eligibility check and only validate page-plan metadata for real context-parallel EXTEND or shared-KV draft extend forwards. Add a regression that TARGET_VERIFY with no prefix metadata returns no-split instead of failing.

Constraint: Absorb syh 7fea88278 only; MQA logits chunk and per-forward budget changes are intentionally excluded.
Rejected: Relax the validator globally | real prefill page-plan violations must remain fail-fast.
Confidence: high
Scope-risk: narrow
Directive: Do not run CP shared-KV page-plan validation for non-context-parallel forward modes without proving those modes own extend_prefix_lens_cpu.
Tested: Remote g0034 container py_compile for utils/test file.
Tested: Remote g0034 container pytest test_nsa_cp_utils.py -k can_cp_split: 8 passed, 92 deselected.
Not-tested: Full ETE speculative non-deepep MoE path.
2026-06-09 19:55:22 +08:00
laoyao0822 81eb138a26 Avoid CP HiCache owner-lane planning stalls during load-back
CP HiCache load-back eviction planning previously recomputed per-owner page counts from node token tensors while scanning evictable leaves. Under shared-KV pressure this can put scheduler-side planning onto an expensive tensor-padding path and stall before load_back can complete.

This stores per-CP-size owner page counts on CP HiCache metadata and uses that CPU metadata for backed/resident CP nodes. Streaming abort handling also accepts int-like status codes so abort responses do not crash on .name/.value access. Temporary debug runbooks remain ignored.

Unnecessary prefill hot-path timing logs were removed before commit; owner-lane eviction now keeps warning-level output for slow planning, insufficient eviction, or remaining deficits only.

Constraint: CP shared-KV cache residency is page-owner based and already records page owners in CpHiCacheNodeMetadata.
Rejected: Keep verbose prefill/owner-lane timing logs | they proved the issue but add hot-path noise after validation.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce tensor-derived owner counting on CP HiCache backed nodes without measuring scheduler CPU/GPU sync cost.
Tested: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py python/sglang/srt/entrypoints/openai/serving_base.py python/sglang/srt/entrypoints/openai/serving_chat.py python/sglang/srt/entrypoints/openai/serving_completions.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py
Tested: git diff --check
Not-tested: Local pytest collection is blocked by missing starlette dependency.
Not-tested: Full ETE after log cleanup; previous pre-cleanup ETE replay reached 136098.82 prompt tok/s without killing prefill.
2026-06-09 06:55:39 +08:00
laoyao0822 50fde834ae Keep CP compute padding out of sparse MoE
CP shared-KV compute padding creates per-request lane slots, so valid rows are not a simple prefix/suffix mask. DeepEP MoE was still seeing dummy rows and using scalar non-padded semantics, which let padding participate in gate/topk and corrupted cache-hit tiny-extend inference.\n\nThe fix compacts CP-local valid rows before MoE dispatch and restores the compact output back to the compute-padded row layout before downstream layer communication. The local GSM8K investigation ledger is now removed from the tracked tree and ignored so future debug notes stay local.\n\nConstraint: CP shared-KV compute-padding layout must keep downstream communicator shapes stable.\nRejected: Disable bs>1/current reuse/cache-hit fast paths | hides the semantic bug and loses the intended performance path.\nRejected: Use num_token_non_padded for MoE under compute padding | valid rows are interleaved with dummy lane slots, not suffix-padded.\nConfidence: high\nScope-risk: moderate\nDirective: Do not feed compute-padded dummy rows into sparse MoE gate/topk; compact valid rows at the MoE boundary and restore shape afterward.\nTested: python -m py_compile python/sglang/srt/layers/attention/nsa/utils.py python/sglang/srt/models/deepseek_v2.py\nTested: remote focused CP utils tests passed, 4 tests.\nTested: remote GSM8K 50-question smoke accuracy 0.960; 200-question runs accuracy 0.955 and 0.965; full 1319-question run accuracy 0.952.\nNot-tested: Long-running production traffic beyond GSM8K after this commit.
2026-06-09 01:48:24 +08:00
laoyao0822 4d2fdd14ac Keep kernel guard aligned with deployment env
The SYH DeepGEMM pick raised the global sglang-kernel runtime guard to 0.4.3, but this deployment and pyproject intentionally use sglang-kernel 0.4.0 while DeepGEMM compatibility is handled by the deep_gemm_wrapper import/call path and tai-provided runtime pieces. A global 0.4.3 guard blocks startup before those targeted checks can run.\n\nConstraint: Do not update pyproject.toml because local and remote environments intentionally differ.\nRejected: pip install sglang-kernel --force-reinstall | mutates the remote environment and conflicts with the pinned project environment.\nRejected: keep a global 0.4.3 guard | blocks the current CUDA deployment despite pyproject requiring 0.4.0.\nConfidence: medium\nScope-risk: narrow\nDirective: Do not raise this global guard without also updating the environment contract and pyproject together; feature-specific wheel/API checks belong near their import/call sites.\nTested: local py_compile for python/sglang/srt/entrypoints/engine.py; remote py_compile in g0034 container; remote installed sglang-kernel version confirmed as 0.4.0.\nNot-tested: full launch_server restart after the guard change.
2026-06-09 00:45:54 +08:00
laoyao0822 eabe0bbfad Preserve CP slot-remap reference paths after SYH pick
The SYH per-request page_inverse migration fixes bs>1 cache contamination, but the conflict resolution made several helper/reference paths require explicit request ids and broke existing unit coverage. This keeps production bs>1 callers on explicit req-id routing while allowing bs=1/reference helpers to infer or default request ids without reintroducing the hot-path batch-global inverse.

Constraint: Runtime bs>1 materialize paths must route through per-request page_inverse rows to avoid cross-request KV aliasing.

Constraint: The remote test container may still have single-row tai-kernel materialize helpers while production bs>1 requires the new req-id ABI.

Rejected: Revert to batch-global page_inverse | it is the GSM8K cache-hit corruption root cause.

Rejected: Update tests only | the helper API is still useful for bs=1/reference callers and py-level regression coverage.

Confidence: medium

Scope-risk: moderate

Directive: Do not remove explicit loc_req_id/current_req_id from production bs>1 call sites; default inference is for legacy/reference use only.

Tested: Local py_compile for cp_shared_kv_runtime.py and nsa_indexer.py.

Tested: Remote container py_compile for cp_shared_kv_runtime.py.

Tested: Remote container pytest: test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_layout.py => 263 passed, 5 warnings, 2 subtests passed.

Not-tested: Full ETE GSM8K/cache-hit run after SYH pick.
2026-06-09 00:19:15 +08:00
laoyao0822 430ed85083 Record CP cache-hit GSM8K findings and cleanup state
The bs>1 warm-cache accuracy investigation produced several negative results, reproducer details, and the final duplicate-logical-page slot-aliasing hypothesis. Keeping the ledger prevents repeated log archaeology after context compaction and documents which probes were removed from runtime code.

Constraint: The document is an investigation ledger, not active runtime instrumentation
Rejected: Drop the tensor-dump history entirely | it explains why later work focused on request-slot remap instead of decode transfer
Confidence: medium
Scope-risk: narrow
Directive: Treat historical dump sections as evidence only; do not infer that dump hooks are still present in runtime code
Tested: Documentation-only; runtime verification covered by the adjacent code/test commits
Not-tested: Rendered documentation formatting outside plain Markdown
2026-06-08 20:57:30 +08:00
laoyao0822 a32c639147 Remove temporary CP cache-hit trace instrumentation
The GSM8K/cache-hit debugging pass added request-correlation logs across scheduler, radix, HiCache, Mooncake, and prefill handoff. The root cause work has moved to request-slot remap semantics, so those high-cardinality traces are no longer needed in the runtime diff.

This removes the temporary tracing helpers and call sites while leaving existing fail-fast checks, fallback warnings, and explicit debug/timing infrastructure that is still part of normal CP shared-KV diagnostics.

Constraint: Production CP hot paths should not carry investigation-only request signatures or transfer summaries
Rejected: Keep all debug logs gated by env | even gated logs increase maintenance surface and encourage stale diagnosis paths
Confidence: high
Scope-risk: moderate
Directive: Reintroduce request-correlation logs only as a narrow opt-in probe with a planned removal point
Tested: Local py_compile for touched runtime files; remote py_compile; remote pytest test_cp_shared_kv_runtime.py, test_nsa_cp_utils.py, test_cp_shared_kv_layout.py => 263 passed, 5 warnings, 2 subtests passed
Not-tested: Long-running ETE log-volume comparison after removal
2026-06-08 20:57:08 +08:00
laoyao0822 3698b0c22c Lock bs>1 CP compute-padding invariants
The GSM8K warm-cache investigation exposed several tiny-extend shapes where compute padding, valid-row selection, last-token collect, and all-gather rerange must stay request-slot aware. These tests pin those invariants without changing runtime behavior.

Constraint: bs>1 tiny extends use page-granular compute slots while only a subset of rows are semantically valid
Rejected: Rely on scalar non-padded token counts for these layouts | valid rows are not always a simple suffix mask
Confidence: medium
Scope-risk: narrow
Directive: Do not weaken these tests unless the replacement path proves equivalent request-slot and valid-row semantics
Tested: Remote pytest test_nsa_cp_utils.py as part of the 263-test CP regression suite
Not-tested: CUDA graph paths; prefill does not use cuda graph in the current CP setup
2026-06-08 20:56:48 +08:00
laoyao0822 b58513cba4 Preserve request-slot identity for CP cache-hit reuse
Warm-cache bs>1 requests can carry duplicate logical page ids across request rows. A batch-global page inverse collapses those request slots and can alias current/prefix KV across requests.

The runtime now carries compact row-scoped sorted page descriptors and remaps flattened logical locs with request row ids where needed, while retaining the legacy global inverse for no-row-context paths.

Constraint: Avoid the rejected dense [batch, logical_page_capacity] inverse because bs up to 10 and long contexts make that memory cost unacceptable
Rejected: Keep global page_inverse for bs>1 duplicate pages | it is lossy and matches the GSM8K warm-cache corruption shape
Rejected: Allocate page_inverse_by_row | correctness-safe but too much GPU memory for production
Confidence: medium
Scope-risk: moderate
Directive: Any future TAI materialize fast path for bs>1 duplicate pages must consume row-scoped descriptors or an equivalent request-row key
Tested: Local py_compile for touched runtime files; remote py_compile; remote pytest test_cp_shared_kv_runtime.py, test_nsa_cp_utils.py, test_cp_shared_kv_layout.py => 263 passed, 5 warnings, 2 subtests passed
Not-tested: Full GSM8K ETE after this cleanup pass
2026-06-08 20:56:31 +08:00
laoyao0822 fb5ccaff26 Preserve CP shared-KV cache-hit diagnostics
Warm-cache GSM8K failures needed request-to-log correlation across scheduler, prefill transfer, and Mooncake CP filtering. The added diagnostics are gated by existing CP shared-KV debug envs and include bounded token signatures plus transfer page summaries so future debugging can identify whether a failed request hit L1, L2, or transfer truncation paths.\n\nThe findings document records the ruled-out hypotheses and the RAGGED current-row contract failure, preventing repeated log archaeology after context compaction.\n\nConstraint: Production hot paths must not emit these logs unless SGLANG_CP_SHARED_KV_BS_GT1_DEBUG or existing CP shared-KV debug gates are enabled.\nRejected: Add a new debug env | reuse of existing bs>1 debug gate avoids more runtime switches.\nRejected: Store full token ids in logs | bounded signatures are enough for correlation without huge logs.\nConfidence: medium\nScope-risk: narrow\nDirective: Keep these diagnostics gated and bounded; do not convert them to unconditional INFO logs.\nTested: Remote container py_compile as part of the RAGGED fix validation.\nNot-tested: Long-running production log-volume impact with debug enabled.
2026-06-08 03:39:50 +08:00
laoyao0822 2aa0b7313e Fix RAGGED CP cache-hit current KV reuse
FP8 flashmla_sparse uses flattened RAGGED page tables that include both cached prefix and the just-computed current suffix. The old cache-hit path materialized the whole flattened range from persistent KV, which could read current rows through the wrong contract under CP shared-KV and compute padding.\n\nThis change makes the RAGGED path use the page-slot partial-current compose contract: prefix pages are materialized from cache slots while current rows are sourced from fresh k/k_rope and packed for FP8 when needed. A new helper accepts the actual current-row contracts seen by attention code: already-local valid rows, CP-local compute-padded rows, or unsplit global valid rows.\n\nConstraint: CP shared-KV stores and consumes cache at page granularity, while attention current rows may be valid-token tensors rather than cache-write local compute rows.\nRejected: Full materialize prefix+current from persistent KV | it can read current suffix through stale or unordered persistent cache state.\nRejected: Reuse select_cp_local_valid_rows_for_cache_write directly | it only accepts CP-local compute-padded rows and killed prefill when RAGGED supplied global valid rows.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not route RAGGED FP8 cache-hit current suffix through full materialize without proving current write ordering and row-contract compatibility.\nTested: Local py_compile for touched runtime/test files.\nTested: Remote container pytest for RAGGED current compose and compute-padding global-current row selection.\nNot-tested: Full GSM8K warm-cache ETE after restart.
2026-06-08 03:39:34 +08:00
laoyao0822 b324407def Trace CP HiCache cache-hit state for GSM8K regressions
The cache-hit accuracy drop is not isolated to one obvious kernel boundary, so the scheduler/radix/HiCache path needs request-correlated evidence.  Add default-off, rate-limited CP shared-KV debug logs for prefix matching, valid-tail flooring, pending backup attach/commit, load-back planning, and unfinished-request cache insertion.  The investigation notes capture current GSM8K pass1/pass2 evidence and rejected interpretations to avoid repeated log archaeology.

Constraint: Logs must be default-off and bounded because these paths are scheduler/control hot paths.

Rejected: Always-on INFO tracing | would distort the exact latency and cache-hit behavior under investigation.

Confidence: medium

Scope-risk: moderate

Directive: Remove or keep env-gated only after the GSM8K cache-hit root cause is closed; do not leave unbounded hot-path logs.

Tested: python -m py_compile on changed runtime files.

Not-tested: Local pytest blocked before collection by missing orjson dependency; fresh restarted GSM8K verification still pending.
(cherry picked from commit bfc6d1473dc2a5d72bc3a8d6fca1e2429537be0e)
2026-06-08 00:17:28 +08:00
laoyao0822 ff4717c2fb Keep L1 free-room eviction off the CP load-back hot path
CP load-back already handles exact owner-lane deficits before admitting L2->L1 reloads.  Treating free-room target misses as synchronous evict work adds avoidable latency to cache-hit requests even when the current load can fit.  The load-back path now records the free-room miss as advisory and leaves proactive replenishment to non-critical eviction paths.

Constraint: GSM8K/cache-hit runs are sensitive to CPU/control-path latency during L2->L1 reload.

Rejected: Evict to free-room on every load-back | preserves room but puts synchronous eviction on the latency-sensitive cache-hit path.

Confidence: medium

Scope-risk: moderate

Directive: Exact owner-lane deficits remain mandatory; free-room targets must not become blocking in load-back without ETE latency evidence.

Tested: python -m py_compile on changed runtime files.

Not-tested: ETE latency after this isolated change not rerun in this commit step.
(cherry picked from commit 2f42acba0d748d2d747f0531963267d86c483cde)
2026-06-08 00:17:28 +08:00
laoyao0822 8f23b06427 Lock fp8 CP cache-hit roundtrips across batch requests
The GSM8K cache-hit regression needs executable coverage for the fp8 paths that combine bs>1 page planning, valid-row writes, prefix materialization, and HiCache L2 reload.  These tests construct bs=5 page-aligned layouts and assert valid rows survive the persistent-page store/materialize and host roundtrip paths.

Constraint: Production runs use fp8_e4m3 CP shared KV with page_first_direct HiCache and bs>1 prefill.

Rejected: Validate with bf16-only tests | the observed regressions are fp8/cache-hit sensitive and bf16 coverage does not exercise scale/index byte layouts.

Confidence: medium

Scope-risk: narrow

Directive: Keep fp8 cache-hit tests aligned with the page-as-minimum-cache-unit contract.

Tested: python -m py_compile on changed runtime files.

Not-tested: CUDA/TAI tests not run locally; local pytest blocked before collection by missing orjson dependency.
(cherry picked from commit 6600f75fc8ee803610df0078bb880df808655635)
2026-06-08 00:17:28 +08:00
laoyao0822 0f9b445131 Keep compute-padding source offsets out of valid rerange output
Batch in-seq CP rerange has two lengths under compute padding: source payload length includes synthetic rows used only to keep CP compute well-shaped, while output length must expose only valid request rows.  The torch fallback now computes rank-major source offsets from compute splits and output placement from valid splits.

Constraint: Tiny extend batching can add compute-padding rows that must not appear in restored valid token order.

Rejected: Use valid splits for source offsets | following requests on the same rank are shifted when a previous request has padded mirror rows.

Confidence: medium

Scope-risk: narrow

Directive: Batch rerange implementations must distinguish source compute splits from valid output splits.

Tested: python -m py_compile on changed runtime files.

Not-tested: Local pytest blocked before collection by missing orjson dependency.
(cherry picked from commit 31e741477503caa52f3a23acdc1286f46079043c)
2026-06-08 00:17:28 +08:00
laoyao0822 c9a39ccdd2 Preserve decode suffix KV locations after cache hits
Decode queue compaction receives req_to_token rows after the prefill side has already populated cached prefix slots.  Cache-hit requests therefore need the extend/suffix slice, not the leading prefix slice, when building the prebuilt transfer chunk.

Constraint: Prefill/decode disaggregation shares req_to_token rows across cached prefix and new suffix positions.

Rejected: Keep slicing from zero | cache-hit requests would copy prefix KV locs into the prebuilt suffix chunk.

Confidence: medium

Scope-risk: narrow

Directive: Do not change prepare_for_prebuilt slicing without testing cache-hit req_to_token layouts.

Tested: python -m py_compile on changed runtime files.

Not-tested: Local pytest blocked before collection by missing orjson dependency.
(cherry picked from commit 416112b617fabe71e8cff7484794af73f3e84440)
2026-06-08 00:17:28 +08:00
laoyao0822 f75ffff8d9 Protect CP shared-KV cache-hit correctness under batched FP8 reuse
Cache-hit GSM8K regressions only appeared after the second pass reused request-specific suffix pages, so this change adds fail-fast transfer validation, masks stale rectangular page-table tails, and extends CUDA/unit coverage across FP8 CP shared-KV write, load, top-k, and materialization paths. The temporary ledger records eliminated hypotheses to prevent re-debugging the same L2 and persistent-cache paths.\n\nConstraint: CP shared KV stores physical pages but scheduler-visible semantics must remain valid-token/page-bounded.\nConstraint: bs>1 FP8 prefill must preserve existing CP shared-KV fast paths without silent fallback.\nRejected: Blame raw HiCache L2 load without tests | L2 KV and index backup/load/materialize roundtrips pass on remote CUDA.\nRejected: Disable current/partial reuse broadly | hides the cache-hit contract regression and costs performance.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not weaken CP shared-KV fail-fast or rectangular-tail masking without rerunning second-pass cache-hit accuracy tests.\nTested: remote CUDA pytest for fused FP8 MLA store, fused persistent index store, L2-loaded FP8 KV materialize, L2-loaded index materialize, ragged top-k offset, TAI batched index MQA prepare.\nTested: local py_compile for touched test files and git diff --check.\nNot-tested: full second-pass GSM8K accuracy after these diagnostic tests; root cause remains under investigation.
2026-06-07 13:26:49 +08:00
laoyao0822 b17976b60d Keep CP HiCache free room from collapsing under cache reuse
Repeated GSM8K/cache-hit traffic can drain a CP owner lane while exact
allocation still succeeds. This changes L1/L2 free-room logic so the trigger
accounts for the pending allocation, then evicts a minimum target chunk or the
exact shortage, whichever is larger. CP load-back also performs best-effort
owner-lane free-room eviction before admitting tiny cache hits that would
otherwise skew one lane.

The commit also adds gated bs>1 prefill timing markers around
recv/bootstrap/batch/inflight polling so future hangs expose the scheduler
boundary instead of only surfacing as a watchdog shutdown. The post-fix repeat
GSM8K failure is recorded as an active regression to continue investigating,
not as old-process noise.

Constraint: Free-room policy must reduce repeated eviction and owner-lane starvation without reserving required+target pages after every allocation
Rejected: Evict to required+target availability | wastes L1/L2 residency under many short cache hits
Rejected: Treat free-room misses as fatal on load-back | exact capacity should remain the strict admission condition
Confidence: medium
Scope-risk: moderate
Directive: Do not remove the gated bs>1 timing markers until repeat-GSM8K hangs have a direct failing boundary
Tested: git diff --check
Tested: python -m py_compile for touched Python files
Tested: remote earlier test_cp_shared_kv_layout.py and test_cp_hicache_metadata.py passed 157 tests after this free-room formula
Not-tested: Two full repeat GSM8K runs after this commit; latest repeat still killed prefill and requires follow-up root-cause work
2026-06-07 01:23:34 +08:00
laoyao0822 68defa3923 Expose CP shared-KV batch timing evidence
The bs>1 path regressed from the beginning of a run, so log throughput alone is not enough to identify whether scheduler preparation, batch splits, index/top-k, sync compose, HiCache load planning, or valid-row selection is dominating. Add a separate env-gated timing channel with rate and slow-threshold controls so production-like runs can identify the hot operation without enabling verbose structural debug logs.\n\nInstrumentation is intentionally observational: it does not change batch planning, cache residency, transfer semantics, or fast-path eligibility. The timing logs are limited and can be filtered by SGLANG_CP_SHARED_KV_BS_GT1_TIMING_SLOW_MS.\n\nConstraint: Need runtime evidence without starting traffic from the agent side.\nConstraint: Existing bs=1 batch-plan behavior remains expected and unchanged.\nRejected: Reuse SGLANG_CP_SHARED_KV_BS_GT1_DEBUG | existing debug logs are structural and already too noisy for timing diagnosis.\nRejected: Add unconditional timing logs | would perturb the hot path and flood long ETE runs.\nConfidence: high\nScope-risk: narrow\nDirective: Keep this timing channel observational; do not use it to gate behavior or silently change batch policy.\nTested: g0034 docker py_compile for changed Python files.\nTested: g0034 docker PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py — 241 passed, 5 warnings, 2 subtests passed.\nNot-tested: Live ETE timing log interpretation under user-driven traffic.\nNot-tested: Nsight correlation with timing events.
2026-06-06 14:28:26 +08:00
laoyao0822 81bfd4bfd3 Reduce CP shared-KV batch overhead without reverting bs1 planning
CP shared-KV bs>1 exposed two separate overhead sources: HiCache load-back could thrash near capacity, and partial-current sync compose could all-reduce row-major page-table gaps between request prefixes. Keep the intended batch-plan path for bs=1, but make host/L1 free-room handling less reactive and teach the MLA/index sync compose path to use exact per-request prefix slot spans instead of one bounding span.\n\nThe exact-span path preserves the single-span IPC fast path for bs=1/single-span cases, while avoiding over-communication for heterogeneous cache-hit batches. The HiCache metadata tests cover host/L1 free-room propagation and load-back batching behavior.\n\nConstraint: bs=1 using CPSharedKVBatchPlan is the expected steady-state path and must not be treated as a regression.\nConstraint: Remote production-like validation runs inside g0034 container /sgl-workspace/sglang-tai.\nRejected: Disable batch-plan for bs=1 | user confirmed this is intended behavior and it would hide the actual bs>1 overhead.\nRejected: Keep one bounding prefix span for bs>1 | row-major page tables can include current/gap slots and inflate per-layer all-reduce work.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not replace exact prefix spans with a single row-major bounding span unless ETE data proves collective launch count dominates gap over-communication.\nTested: g0034 docker py_compile for changed Python/test files.\nTested: g0034 docker PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py — 241 passed, 5 warnings, 2 subtests passed.\nNot-tested: Full ETE bs>1 throughput after the exact-prefix-span change.\nNot-tested: CUDA kernel benchmark and live traffic replay.
2026-06-06 02:03:16 +08:00
laoyao0822 1b99de7459 Expose gated evidence for CP shared-KV bs>1 debugging
The bs>1 prefill path has multiple coupled stages: scheduler admission, page-aligned batch planning, tensor splitting, direct cache writes, index top-k, MLA reuse, and disaggregated KV handoff. Add a default-off, rate-limited debug channel so production ETE runs can identify where batching or metadata semantics diverge without permanently increasing hot-path log volume.

Constraint: Logs must be default-off and rate-limited because these paths execute per-rank and often per-layer.

Rejected: Always-on INFO logs | would flood logs and add CPU overhead during normal prefill.

Rejected: Only scheduler-side logging | insufficient to distinguish planner, index, MLA, and transfer handoff failures.

Confidence: medium

Scope-risk: moderate

Directive: Keep bs>1 debug evidence env-gated; do not add unconditional per-layer or per-token logs in these paths.

Tested: Local py_compile for touched files

Tested: git diff --check

Tested: Remote py_compile and targeted NSA CP utility tests: 5 passed

Not-tested: Full ETE correctness with debug disabled
2026-06-05 23:48:08 +08:00
laoyao0822 6eea77e5e9 Stabilize disaggregated decode burst handoff
Burst decode can sit in disaggregation prealloc/transfer queues while health probes still need an immediate scheduler-alive response, and EAGLE prebuilt metadata must outlive the synthetic prebuilt step until the first real decode consume. The transfer path also needs to send final aux metadata even when there are no new KV pages in the final chunk.

This keeps decode metadata slot ownership narrow instead of cloning EAGLE tensors, adds fail-safe cleanup for prebuilt exceptions/finished prebuilt requests, fixes final empty chunk metadata transfer, and records the investigation ledger for future debugging.

Constraint: bs=1 historically worked, so decode compute/sampling behavior must not be changed without direct evidence.

Rejected: Clone EAGLE metadata tensors on transfer commit | avoids lifetime issues but adds hot-path CPU/GPU memory traffic.

Rejected: Treat prefill AbortReq logs as root cause | current evidence shows they can be downstream of decode/router aborts.

Confidence: medium

Scope-risk: moderate

Directive: Do not move EAGLE metadata slot release earlier than first real decode result processing without proving the H2D/spec_info consume point changed.

Tested: g0034 docker py_compile for touched scheduler/disagg/test files; PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/mem_cache/test_req_to_token_pool.py test/registered/unit/managers/test_scheduler_health_check.py -> 24 passed

Not-tested: Fresh end-to-end burst traffic after restart; decode node_rank=1 persistent log capture.
2026-06-05 19:59:04 +08:00
laoyao0822 5eecc5f9c5 Preserve CP ragged topk coordinates under batch planning
Batch-planning metadata can now be attached even for single-request CP prefill, which routed FP8 flashmla_sparse through the batch cp_index path. That path used compact MQA-buffer row bases for score lookup but did not override the final ragged topk coordinate base consumed by attention, so topk indices could point at the wrong KV rows and produce low accept length or meaningless output.\n\nThis keeps ordinary long page tails out of compute padding, reserves compute padding for truly tiny suffixes, and makes cp_index RAGGED topk emit request-ragged offsets while preserving the compact buffer descriptors used for score materialization. The debug ledger records the rejected intermediate diagnoses and the confirmed coordinate-space failure.\n\nConstraint: CP shared-KV cache residency is page-granular, but attention/index compute must not consume synthetic long-tail rows.\nConstraint: FP8 CP prefill uses flashmla_sparse/RAGGED, where fused topk output is consumed directly as attention page_table_1.\nRejected: Disable current reuse or batch planning | would hide the regression and lose the intended bs>1 fast path.\nRejected: Treat all page tails as compute padding | regresses bs=1 semantics and can corrupt query/topk row contracts.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not change cp_index RAGGED topk offset handling without verifying score-buffer row bases and final attention KV coordinate space independently.\nTested: python -m py_compile on touched Python/test files; git diff --check; remote targeted ragged cp_index offset regression test; remote test_nsa_cp_utils.py; remote test_cp_shared_kv_runtime.py; user-reported ETE output recovered after restart.\nNot-tested: Agent-driven full ETE traffic run; broad multi-request bs>1 production soak.
2026-06-05 07:45:29 +08:00
laoyao0822 c8510938b9 Repair CP current-slot composition instead of disabling reuse
CP shared KV current-only and partial-current paths were previously unsafe because owner-local current rows could be filled into dense page slots without synchronizing the current slots across CP ranks. This preserves the fast path and fixes the missing synchronization at the actual contract boundary: prefix slots remain materialized by the existing IPC/collective path, while freshly produced current slots are composed from current forward tensors and reduced only over the current slot ranges.\n\nThe change also keeps page-tail compute padding explicit, restores current-only index compose, fixes TAI index descriptor dtype consistency, and records the investigation ledger to avoid repeating discarded hypotheses.\n\nConstraint: CP shared KV must preserve current reuse and TAI materialize fast paths; blanket disable is not an acceptable fix.\nConstraint: Prefix slots must not be reduced twice after IPC/prefix materialization.\nRejected: Disable current-only MLA/index reuse | hides the owner-local composition bug and regresses the intended fast path.\nRejected: Disable TAI materialize globally | avoids symptoms without proving the byte/layout contract.\nConfidence: medium\nScope-risk: broad\nDirective: Do not remove current-slot range synchronization unless replacing it with an equivalent owner-aware P2P/IPC gather contract.\nTested: Local py_compile for touched files.\nTested: Remote g0034 py_compile for touched runtime/test files.\nTested: Remote test_cp_shared_kv_runtime.py: 111 passed, 5 warnings, 2 subtests passed.\nTested: Remote targeted current-slot compose tests and direct current-only index compose script.\nNot-tested: Full ETE output quality and decode accept len after restart.\nNot-tested: Full test_nsa_cp_utils.py collection on remote due incomplete installed sgl_kernel import/fake-op environment.
2026-06-05 06:25:06 +08:00
laoyao0822 e200091638 Expose CP shared-KV transfer page-count mismatches before decode
Mooncake previously truncated source prefill pages when the selected decode destination page list was shorter. That made an invalid prefill/decode page mapping continue as an incomplete KV transfer, which can surface later as decode garbage instead of the original mapping error.\n\nThis changes the transfer contract to require exact source/destination page-count equality and records compact page summaries in the fail-fast error. The helper is shared so the page-count contract can be unit-tested without constructing the transfer worker thread.\n\nConstraint: CP shared-KV page ownership requires a one-to-one prefill source page to decode destination page mapping.\nRejected: Keep warning-and-truncate | masks page-map corruption and can silently drop KV.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not restore source-page truncation; fix the mapping producer if this fail-fast triggers.\nTested: git diff --check; python -m py_compile python/sglang/srt/disaggregation/utils.py python/sglang/srt/disaggregation/mooncake/conn.py test/registered/unit/disaggregation/test_cp_shared_kv_transfer_mapping.py\nNot-tested: pytest blocked locally by missing orjson; remote g0034 was unavailable during this pass.
2026-06-04 23:14:22 +08:00
laoyao0822andOmX f50e2b1e00 Prevent stale CP shared-KV contracts from corrupting prefill
CP shared-KV now uses CP-local current rows consistently across MLA/index current reuse, passes fp8 current-index K through the tai-kernel uint8 ABI, and clears the transient EAGLE CP-local hidden marker after draft capture. The disaggregation bootstrap also fingerprints the runtime source contract so prefill/decode mismatches fail fast instead of silently exchanging incompatible KV metadata.

Constraint: CP shared-KV batch paths flatten current K/V rows in CP-rank-local valid order, not global request order.

Constraint: tai-kernel current-index prepare validates current_index_k as uint8 bytes for fp8 payloads.

Rejected: Keep using global extend offsets for bs>1 current-index reuse | corrupts request-local bases once current_index_kv is CP-local.

Rejected: Infer CP-local EAGLE hidden semantics from tensor shape | static padding and bs>1 can make shape-based inference unsafe.

Confidence: medium

Scope-risk: moderate

Directive: Do not reintroduce forward_batch.out_cache_loc slicing in CP shared-KV current reuse without verifying CP-local owner-lane layout.

Tested: Remote container py_compile for touched runtime/test files.

Tested: Remote PYTHONPATH=python pytest -q test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/disaggregation/test_common_conn_runtime_fingerprint.py (198 passed, 2 subtests passed).

Not-tested: Full remote ETE traffic after this commit; accept length and garbage-output recovery still require a fresh prefill/decode run.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-04 20:22:29 +08:00
laoyao0822 3d6007246b Stabilize CP shared-KV batch padding semantics
CP shared-KV bs>1 exposed three distinct padding domains: valid cache rows, CP page-tail compute rows, and MLP-sync flattened static padding. The previous implementation mixed these domains in direct-write and index top-k paths, so real requests failed when q/out_cache_loc lengths matched valid rows while metadata aliases described compute rows.\n\nThis change makes compute split strip only proven flattened static padding, keeps valid cache writes strict except for extend_num_tokens-proven static tails, marks CP-local EAGLE draft hidden state explicitly, and selects NSA index top-k query metadata by the actual q/weight row count.\n\nConstraint: CP shared-KV cache writes must never persist dummy page-tail or MLP static padding rows.\nConstraint: EAGLE draft hidden state can be CP-local before full CP metadata is visible in prepare_mlp_sync_batch.\nRejected: Use compute_padding_enabled as direct-write truncation proof | it silently accepts unknown out_cache_loc tails.\nRejected: Always consume compute q metadata in index top-k | actual q/weights can be valid-only after CP split.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not collapse valid rows, CP compute padding, and MLP static padding into one length condition; use explicit provenance.\nTested: remote py_compile for touched NSA files\nTested: remote targeted CP shared-KV padding/top-k regressions\nTested: remote pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py -k 'not test_tai_current_slot_fill_sparse_page_self_test_passes_on_installed_kernel' => 228 passed, 1 deselected, 5 warnings, 2 subtests passed\nNot-tested: full ETE replay after the final index top-k fix\nNot-tested: TAI current-index fast path dtype fallback
2026-06-04 07:25:11 +08:00
laoyao0822 02af370e87 Batch CP HiCache host admission before write reservation
CP shared-KV prefill batching can present multiple requests to HiCache write-through in one forward. Keeping host free-room admission per request causes repeated small L2 evictions and defeats the free-room policy, while request metadata and radix attachment still need per-request ownership.\n\nThis change adds a batch prepare path that aggregates required host tokens by CP owner lane, performs one host admission/eviction step for the batch, and then reserves/submits each request independently. SessionAwareCache and the scheduler prefer the batch API when available.\n\nConstraint: Radix nodes, host slots, draft slots, rollback, and ack semantics remain per request.\nRejected: Merge reservations or radix nodes across requests | would complicate rollback and writing_check ack ownership.\nRejected: Add collective synchronization for host admission | local owner-lane logic already provides deterministic capacity accounting.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce per-request host free-room eviction in the scheduler path without profiling host-full workloads.\nTested: local py_compile for touched Python files\nTested: remote g0034 docker PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 117 passed, 5 warnings\nNot-tested: full ETE bs>1 HiCache run under production traffic
2026-06-04 04:32:40 +08:00
laoyao0822 108e7d866d Gate CP shared-KV prefill batching behind explicit limits
The scheduler can now admit multi-request NSA in-seq CP shared-KV prefill batches only when the shared-KV bs>1 flag is explicitly enabled. The gate is still disabled by default and is scoped to CP shared-KV so ordinary CP is not widened accidentally.

Batch admission is bounded by optional request-count and page-aligned extend-token limits while real memory capacity remains allocator-owned. This keeps bf16 and fp8 on the same scheduler path because dtype differences are already reflected in KV pool token/page capacity.

Constraint: bs>1 runtime paths remain guarded by existing CP shared-KV fail-fast checks.

Constraint: Scheduler must not duplicate bf16/fp8 byte-level capacity estimation.

Rejected: Open the old CP gate unconditionally | ordinary CP would inherit an unverified shared-KV-specific batching path.

Rejected: Treat the extend-token cap as a hard per-request limit | a single large request could deadlock the scheduler.

Confidence: medium

Scope-risk: moderate

Directive: Keep CP shared-KV batching gated until ETE validates EAGLE accept length, output length, and HiCache load/backup behavior under real traffic.

Tested: local py_compile for server_args, schedule_policy, scheduler, prefill_adder tests, and server_args tests.

Tested: remote g0034 py_compile for the same files.

Tested: remote g0034 pytest target set: 5 passed for parser, parameter validation, default single-request CP gate, enabled bs>1 gate, and page-aligned extend cap.

Tested: remote g0034 pytest test_prefill_adder.py => 13 passed.

Not-tested: full server_args test file has an unrelated HuggingFace DNS/config-download failure in TestPrepareServerArgs.test_prepare_server_args.

Not-tested: ETE production traffic with --enable-cp-shared-kv-prefill-bs-gt1.
2026-06-04 04:17:32 +08:00
laoyao0822 d7723aca07 Prevent batched CP draft from silently leaving the target path
EAGLE draft shared-KV is supposed to mirror the target CP layout, so bs>1 must not fall back to legacy full-input or padded-hidden behavior when required batch metadata is missing or inconsistent. This change keeps the existing bs=1 compatibility path but makes batched CP draft fail fast on missing/mismatched spec hidden states, embedding pad metadata, or input embed shapes. The docs record the current W7 boundary: draft prefill follows target metadata, while scheduler admission and ETE remain gated.

Constraint: CP draft KV must mirror target layout and must not silently diverge under bs>1 shared-KV.

Rejected: Allow bs>1 to use the old full-input fallback | it can hide wrong owner/page metadata and corrupt accept length.

Confidence: medium

Scope-risk: moderate

Directive: Do not open the scheduler bs>1 CP gate until EAGLE accept length/output length are verified with this fail-fast path enabled.

Tested: Remote g0034 targeted EAGLE fail-fast unit test passed; remote full test/registered/unit/layers/test_nsa_cp_utils.py passed 70 tests.

Not-tested: EAGLE bs>1 ETE, because scheduler CP bs>1 admission gate remains closed.
2026-06-04 03:47:56 +08:00
laoyao0822 3e3f1b776b Enable CP shared-KV compute padding without inflating cache state
Tiny extend requests can leave most CP lanes without query work, which has been tied to hangs and accept-length regressions. This change introduces a dual valid/compute metadata contract: forward paths may materialize compute-padded rows, while cache, current reuse, direct write, HiCache backup, and load remain valid/page based.

The implementation keeps radix/HiCache/device allocation on real page extents, filters dummy compute rows before MLA/index cache writes and current reuse, makes top-k/index consume compute rows while compacting valid rows, and opens tiny CP shared-KV in-seq split through compute padding. The accompanying plan document records the contract and P1-P7 evidence.

Constraint: CP shared KV and HiCache must stay page-granular; dummy compute rows must not allocate, write, backup, or load KV cache.

Constraint: Avoid silent fallback and avoid adding collectives on hot paths.

Rejected: Pad cache allocations to cp_size pages | would waste KV capacity and pollute radix/HiCache state.

Rejected: Keep tiny suffixes out of CP split | preserves the zero-lane behavior that compute padding is meant to remove.

Confidence: medium

Scope-risk: broad

Directive: Do not route compute-padded dummy rows into out_cache_loc, current reuse, HiCache reservation, or backup descriptors; keep valid/cache metadata explicit.

Tested: Remote g0034 container targeted P7 tests: 3 passed, 3 warnings.

Tested: Remote g0034 container full unit slice: PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/mem_cache/test_cp_shared_kv_layout.py => 214 passed, 5 warnings, 2 subtests passed.

Tested: Local py_compile for touched P7 test file.

Not-tested: Latest CUDA/ETE traffic validation for dummy top-k rows, accept len, output len, and detokenizer hang behavior.
2026-06-04 01:34:35 +08:00
laoyao0822 b3913046b6 Batch CP HiCache backup submits across requests
CP HiCache write reservations must stay per radix node, but the transfer descriptor does not need to be per request. This changes the layer-end hook to group pending write states for the same source and layer, so bs>1 prefill emits one target D2H descriptor and one draft D2H descriptor per layer while preserving per-node metadata, rollback, and ack semantics.\n\nConstraint: CP shared-KV HiCache metadata, host slots, and radix acknowledgements remain per request/node.\nConstraint: TAI direct transfer kernels already accept flattened page descriptors, so no tai-kernel change is required.\nRejected: Merge HiCache reservations or radix nodes | would complicate rollback and split handling.\nRejected: Add collective synchronization for grouped backup | grouping is local descriptor construction and must not add rank-level sync.\nConfidence: high\nScope-risk: moderate\nDirective: Keep target and draft source notifications separate; final ack must wait for both when draft HiCache is attached.\nTested: local py_compile for cache_controller.py and test_hicache_controller_cp.py\nTested: local git diff --check\nTested: remote pytest test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py => 85 passed, 3 warnings\nNot-tested: full ETE bs>1 CP HiCache replay with admission gate removed\nNot-tested: Nsight/throughput validation of reduced D2H submit count
2026-06-03 10:23:30 +08:00
laoyao0822 19dcd6c4dc Batch CP shared-KV index work for bs>1 fast paths
The bs>1 path needs index top-k, shared-index prepare, current-index compact, and current-slot compose to consume flattened batch descriptors instead of falling back to per-request or per-segment Python/Torch work. This change wires SGLang to the new TAI batch prepare kernels, keeps fallback explicit, and records the remaining HiCache/load-backup gaps in the bs>1 workstream docs.

Constraint: CP shared-KV bs>1 must reuse fast paths rather than adding slow batch-only fallbacks

Constraint: No new collective operations were introduced

Rejected: Leave current-only cp_index as Python slice/cat | it keeps per-segment overhead in the short-extend bs>1 case

Rejected: Infer max segment lengths from CUDA descriptor tensors | .item() would add CPU synchronization on the hot path

Confidence: medium

Scope-risk: moderate

Directive: Do not remove the explicit fallback warnings without verifying the corresponding TAI symbols are present in production

Tested: local py_compile for touched SGLang files

Tested: remote g0034 test_nsa_cp_utils.py passed, 53 tests

Tested: remote g0034 test_fill_current_index_page_slots_uses_tai_kernel_when_available passed

Not-tested: full ETE bs>1 traffic with HiCache load/backup and draft/EAGLE enabled
2026-06-03 08:45:32 +08:00
laoyao0822 d36f62a3cd Merge CP shared-KV batch owner allocation
PR #10 provides the W2 owner-lane allocation side needed by the current bs>1 CP shared-KV work: supported multi-request extends now derive page owners per request, preserve flattened request order, and use owner-lane allocation instead of legacy page allocation.\n\nThe merge applies cleanly on top of the W3/W4 target current-reuse work because the touched runtime surface is allocator/mem_cache only. Focused remote tests cover the new owner-lane planner/allocation behavior and the existing W4 target reuse regressions.\n\nConstraint: CP shared-KV page ownership is request-relative and page-granular.\nConstraint: Supported bs>1 CP shared-KV allocation must not silently fall back to legacy allocation.\nRejected: Treat multiple requests as one concatenated extend | would assign owners relative to the wrong request boundary.\nRejected: Keep multi_batch fallback | breaks downstream direct-write owner assumptions.\nConfidence: medium\nScope-risk: moderate\nDirective: Preserve request-order flattening when changing owner allocation; do not reorder by lane for convenience.\nTested: Local py_compile for allocator.py, common.py, cp_shared_kv_compute_owner.py, and test_cp_shared_kv_layout.py.\nTested: Local git diff --check --cached.\nTested: Remote g0034 py_compile for touched mem_cache files and test_cp_shared_kv_layout.py.\nTested: Remote g0034 pytest test_cp_shared_kv_layout.py => 34 passed, 3 warnings.\nTested: Remote g0034 pytest test_nsa_cp_utils.py test_cp_shared_kv_runtime.py => 157 passed, 5 warnings, 2 subtests passed.\nNot-tested: Full ETE/perf run; production HiCache load/back up with bs>1 allocation under sustained traffic.
2026-06-03 07:22:04 +08:00
laoyao0822 50d0008705 Enable batched CP shared-KV current reuse correctness
Batch-size > 1 cache-hit traffic must not lose the current/partial-current reuse fast path. This change extends the target MLA/index sync-correct path to validate batched current suffix rows, compose page-aligned prefix spans, and route batched index top-k through current-only or partial-current reuse without falling back to scalar guards.\n\nThe implementation keeps page as the minimum cache unit: prefix cache coverage is page-aligned, while current suffix rows are sliced by valid extend lengths so padded tail rows are not exposed to attention. The index top-k batch path now mirrors the single-request contract: current-only reuses current index KV directly, partial-current materializes once and shares the dense buffer across request segments.\n\nConstraint: CP shared-KV supports in-seq zigzag only; round-robin remains fail-fast for shared-KV.\nConstraint: No new collectives are introduced; this is a sync correctness path, not a new communication scheme.\nRejected: Keep bs>1 current_index_kv fail-fast | disables the cache-hit path that W4 is meant to restore.\nRejected: Pad requests to a common batch length | violates the page-granular contract and exposes padded tails to consumers.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce batch_size_not_one or batch_gt1 current-reuse guards without proving an equivalent batched fast path exists.\nTested: Local py_compile for touched runtime/test files.\nTested: Remote g0034 pytest test_nsa_cp_utils.py test_cp_shared_kv_runtime.py => 157 passed, 5 warnings, 2 subtests passed.\nTested: Remote g0034 pytest test_cp_shared_kv_layout.py => 27 passed, 3 warnings.\nNot-tested: Full ETE with live traffic; CUDA kernel-backed batched descriptors; L1 prefetch, HiCache load/backup, and draft/EAGLE bs>1 current reuse.
2026-06-03 07:06:21 +08:00
laoyao0822 a7472c415f Preserve request boundaries in CP shared-KV index top-k
W4-1 needs target index/top-k sync correctness before current/partial-current reuse can be restored. Batch-size>1 in-seq CP produces local q/weights in request-segment order, so top-k must consume req0_prev, req0_next, req1_prev, req1_next rather than treating the flattened batch as one scalar prev/next pair.

The implementation adds a batch dispatch for _get_topk_in_seq_cp_pair, reuses one synchronous shared-index materialization per layer, and calls _get_topk_ragged_with_cp per request segment with an explicit batch_idx. The scalar bs=1 path remains unchanged.

Constraint: This is W4-1 target index/top-k sync correctness; original W4 current/partial-current reuse remains a separate follow-up.

Constraint: Phase W4-1 must not enable bs>1 index prefetch, current reuse, partial-current reuse, or the cp_index multi-batch branch.

Rejected: Use cp_index branch for multi-batch | source marks that path as having accuracy issues.

Rejected: Pad batch requests to max length | wastes compute and violates packed/ragged batch contract.

Confidence: high

Scope-risk: moderate

Directive: Keep bs>1 target top-k ordered by request segment unless a later fused descriptor proves identical ordering and correctness.

Tested: Remote g0034 py_compile for nsa_indexer.py

Tested: Remote g0034 PYTHONPATH=python pytest test/registered/unit/layers/test_nsa_cp_utils.py -> 45 passed

Tested: Remote g0034 PYTHONPATH=python pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py -> 172 passed, 2 subtests passed

Not-tested: Full ETE bs>1 serving run with live traffic
2026-06-03 02:36:31 +08:00
laoyao0822 f8b4f1915e Make CP shared-KV direct writes enforce batch-local ownership
W3 needs batch-size>1 extends to use packed valid tokens while preserving per-request page boundaries. The local out_cache_loc planner now validates against the batch plan's request lengths instead of the first request's scalar split metadata, then reuses the existing batch split helper to produce this rank's logical/physical cache locs.

Direct-write failures inside the CP shared-KV contract now fail fast instead of silently falling back to legacy index/MLA stores. This exposes allocator, owner-lane, page-alignment, and shape-contract bugs early for both bs=1 and bs>1.

Constraint: bs>1 batching must not pad short requests to the longest request; only per-request page-boundary padding is allowed.

Rejected: Keep bs=1 compatibility fallback | it hides CP shared-KV contract violations and caused repeated slow-path ambiguity.

Rejected: Pad batch to max request length | wastes compute and complicates cache validity for short extends.

Confidence: high

Scope-risk: moderate

Directive: CP shared-KV contract errors should stay fail-fast; do not reintroduce silent direct-write fallback without ETE evidence and explicit warning semantics.

Tested: Remote g0034 py_compile for utils.py nsa_indexer.py forward_mla.py

Tested: Remote g0034 PYTHONPATH=python pytest test/registered/unit/layers/test_nsa_cp_utils.py -> 43 passed

Tested: Remote g0034 PYTHONPATH=python pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py -> 170 passed, 2 subtests passed

Not-tested: Full ETE bs>1 serving run with live traffic
2026-06-03 02:13:57 +08:00
laoyao0822 e4cf8d18b4 Preserve CP narrow output while planning real batches
Batch-size support needs request-first CP metadata; treating a batch as one long sequence breaks page ownership, top-k ranges, and phase1 compact output collection. This adds a batch CP plan that records per-request page-aligned splits, rank-local offsets, kv/actual-seq metadata, last-token owners, and flattened descriptors for downstream allocator/runtime workstreams.

The scalar full-rerange path now fail-fasts for batch metadata so bs>1 cannot silently discard the narrow-output optimization or restore hidden states with single-request assumptions.

Constraint: CP shared-KV cache state is page-owned and must preserve request boundaries under bs>1.

Rejected: Let bs>1 fall back to scalar full hidden rerange | it loses the phase1 communication reduction and uses wrong single-request metadata.

Rejected: Add a collective to confirm batch plans | all ranks can derive the same plan from CPU metadata and config.

Confidence: medium

Scope-risk: moderate

Directive: Do not remove batch fail-fast guards until W2/W3 consumers use CPSharedKVBatchPlan end-to-end.

Tested: python -m py_compile python/sglang/srt/layers/attention/nsa/utils.py test/registered/unit/layers/test_nsa_cp_utils.py

Tested: remote g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py -> 39 passed

Not-tested: full ETE bs>1 CP shared-KV runtime; W2/W3 allocator/direct-write consumers are not implemented yet
2026-06-03 01:21:43 +08:00
laoyao0822andOmX 0158e28689 Plan batch-aware CP shared-KV execution
The existing CP shared-KV notes still reflected the earlier multi-slot-first conclusion, while current short cache-hit extends require true ForwardBatch>1 planning. This commit records the batch-aware contract, sequential phase plan, and parallel workstream split so implementation can proceed without rediscovering single-request blockers or accidentally removing guards into unsafe fallback paths.

Constraint: Current NSA in-seq CP shared-KV paths assume batch_size == 1 across metadata, owner-lane allocation, direct write, current reuse, prefetch, top-k, and EAGLE/draft collection.

Rejected: Remove batch_size guards directly | would treat multiple requests as one long sequence and corrupt request boundaries and page-owner state.

Rejected: Keep only the old multi-slot recommendation | online extend sizes around 200-2000 tokens make true batching a required throughput direction.

Confidence: high

Scope-risk: narrow

Directive: Implement bs>1 by per-request page-aligned planning before flattening; do not add planner collectives or silent legacy fallbacks.

Tested: Placeholder scan and referenced-path existence check for all four docs; git diff --cached --check.

Not-tested: Runtime behavior; documentation-only change.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-03 00:37:54 +08:00
laoyao0822 937f89ef89 Avoid index prefetch fallback noise when disabled
The missing index prefetcher state is expected when MLA/index prefetch is disabled. Logging it as an index-prefetch fallback made the normal synchronous materialize path look like a failed fast path and created noisy reports.

This keeps consume-miss warnings for enabled prefetchers, but suppresses missing-prefetcher warnings when the prefetch feature gate is off.

Constraint: Index prefetch is controlled by the same MLA prefetch enable gate in the current pipeline.
Rejected: Suppress all index prefetch fallback warnings | consume misses still indicate an enabled prefetch pipeline failed to serve a requested buffer.
Confidence: high
Scope-risk: narrow
Directive: Missing prefetcher is not a fallback when SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH=0; do not re-add warning noise without a separate enable gate.
Tested: Local py_compile for nsa_indexer.py and test_cp_shared_kv_runtime.py.
Tested: Remote g0034 targeted regression set passed 4 tests: disabled sync materialize no warning, consume miss warning, first-layer no-warning, enabled create-skip warning.
Not-tested: Full CUDA ETE run with logs.
2026-06-02 23:52:46 +08:00
laoyao0822 401de0f8ce Reduce CP HiCache L2 allocator scan cost
Host HiCache reservations were paying token-level free-slot scans when trying to preserve page contiguity. The allocator now keeps a lazy page-extent index so availability checks and contiguous-preferred allocations avoid materializing the full 220GB-equivalent free-slot metadata path.

The companion benchmark models steady-state L2 churn near full occupancy, including burn-in and historical node-size effects, so LPF/RDMA descriptor quality can be separated from ETE noise.

Constraint: CP HiCache host allocations are page-shaped, but existing callers may still read free_slots directly.
Rejected: Sort and scan free_slots on each alloc_contiguous_preferred call | measured ms-level CPU overhead on 220GB-equivalent metadata.
Rejected: Remove free_slots compatibility | storage/tests still rely on the public tensor surface.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce per-allocation full free_slots scans on HostKVCache; preserve page-extent metadata or benchmark before changing allocator shape.
Tested: Local py_compile for memory_pool_host.py, allocator benchmark, and related tests.
Tested: Local test_cp_hicache_allocator_bench.py 10 passed.
Tested: Remote g0034 test_hicache_controller_cp.py 67 passed; test_cp_hicache_allocator_bench.py 10 passed.
Tested: Remote 220GB-equivalent host_churn benchmark: contiguous path reduced from ms-level to ~30-292us p50 depending on fragmentation.
Not-tested: Full CUDA ETE run after allocator change.
Not-tested: Production long-run fragmentation behavior under live traffic.
2026-06-02 23:35:45 +08:00
laoyao0822 7c8fa2f71c Remove full-cache scans from CP owner-lane allocation
The CP shared-KV allocator was still doing total-cache-sized CPU work in the scheduler hot path.  That cannot be hidden by GPU overlap, so owner-lane allocation now maintains per-owner free/release buckets and consumes request-sized prefixes instead of rebuilding masks over the full free-page tensor on each request.\n\nThe benchmark was extended to isolate L1 stats, selection, and allocation costs, and the CPU layout tests now install a complete sgl_kernel stub before importing SGLang helpers so remote unit collection does not abort in native extension loading.\n\nConstraint: Allocator CPU work blocks scheduler progress and cannot overlap with GPU forward execution.\nConstraint: CPU unit tests must not load native sgl_kernel on remote images where the loader can SIGABRT.\nRejected: Keep contiguous-run search over full free_pages | still scales with cache capacity and measured multi-ms overhead.\nRejected: Treat remote collection abort as an environment-only issue | it prevented allocator regression coverage and was fixable with a test-local stub.\nConfidence: high\nScope-risk: moderate\nDirective: CP owner-lane allocation is bucket-based; do not reintroduce full free_pages scans on the hot path without benchmark evidence.\nTested: Local py_compile for touched files\nTested: Local benchmark unit test, 6 passed\nTested: Remote benchmark unit test, 6 passed\nTested: Remote test_alloc_pages_with_owners.py, 10 passed\nTested: Remote test_cp_shared_kv_layout.py, 27 passed\nTested: Remote production allocator microbench shows select/alloc p50 reduced from ms-scale to sub-ms scale\nNot-tested: Full ETE traffic run after allocator bucket change
2026-06-02 08:41:00 +08:00
laoyao0822 ce3a20d11b Stabilize CP HiCache residency under L1/L2 pressure
CP shared KV now keeps explicit L1 and host free-room targets so pressure is handled by planned eviction instead of repeated capacity-edge retries. The host allocator gains contiguous-preferred page reservation, L1 owner-lane allocation prefers contiguous physical pages, and CP HiCache metadata preserves pending backup safety for page-granular radix updates. Mooncake transfer stats and allocator microbenchmarks are included to make the remaining transfer bottlenecks measurable rather than inferred.

Constraint: CP shared KV uses decode CP size 1 with all prefill CP ranks participating in transfer, so L1/L2 cache residency must remain page-granular and avoid extra collectives.\nConstraint: Production HiCache can be hundreds of GB, so allocator metadata overhead must be visible before enabling aggressive contiguous allocation broadly.\nRejected: Evict only the exact deficit | this keeps the cache at the cliff and causes repeated evict/allocate pressure.\nRejected: Rely on allocator scans alone for contiguity | remote microbenchmarks show fragmented 220GB-equivalent host metadata can make contiguous-preferred scans multi-ms.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not increase L1/L2 free-room defaults or add new CP collectives without ETE evidence and transfer/allocator measurements.\nTested: python -m py_compile on touched runtime/test/benchmark files.\nTested: PYTHONPATH=. python -m pytest -q test/registered/unit/benchmark/test_cp_hicache_allocator_bench.py => 4 passed, 1 warning.\nTested: Remote g0034 log /mnt/beegfs/cjy/log/sglang_cp_hicache_20260601_233723.log shows active prefill process with L1/L2 free-room args, 702 HTTP 200 chat completions, 6272 prefill batches, and no fatal scheduler traceback in latest scan.\nTested: User-reported L1/L2 cache ETE validation passed on remote run.\nNot-tested: Full local pytest suite; local environment is missing several runtime dependencies.\nNot-tested: CUDA allocator microbenchmark during active production prefill process.\nNot-tested: Mooncake straggler fix; stats show transfer tail latency remains a separate bottleneck.
2026-06-02 07:58:02 +08:00
laoyao0822 8be4a3a8b5 Surface CP shared-KV fast-path misses
CP shared-KV performance debugging depends on seeing when the runtime leaves the intended TAI, IPC, prefetch, or current-reuse paths. This change makes those misses visible through standardized warning markers while keeping per-reason log limits to avoid per-layer log floods.\n\nThe warnings intentionally distinguish fallback from fail-fast: unsupported correctness-sensitive states still raise, while performance-path misses emit [CP_SHARED_KV_FALLBACK] with the concrete reason.\n\nConstraint: Production ETE debugging needs visible fallback evidence without enabling heavy debug mode, which can itself disable fast paths.\nRejected: Rely only on optional MLA prefetch debug logs | they are env-gated, layer-limited, and miss non-prefetch TAI/IPC/current-reuse fallbacks.\nRejected: Log every per-layer event without limits | would drown useful transfer/cache diagnostics under steady-state traffic.\nConfidence: high\nScope-risk: moderate\nDirective: Do not remove these fallback warnings unless an equivalent low-noise observability path exists for every fast-path miss.\nTested: local py_compile for touched files; local git diff --check for touched files; remote g0034 py_compile and pytest for test_nsa_cp_utils.py, test_cp_shared_kv_layout.py, test_cp_shared_kv_runtime.py passed before commit (156 passed, 5 warnings, 2 subtests passed).\nNot-tested: full ETE serving traffic after warning additions.
2026-06-02 06:47:40 +08:00
laoyao0822 6a25c312c7 Cache completed chunked prefill pages in CP HiCache
Chunked prefill was skipping prepared CP HiCache backup entirely while still inserting unfinished valid-tail radix state. That combination caused middle chunks to miss reusable host cache and repeatedly hit stale-tail split fallback paths.\n\nThis keeps CP HiCache page-granular: middle chunks now reserve/write only completed pages, and unfinished radix inserts expose only page-complete prefixes while preserving the request-owned sub-page tail in prefix_indices for later chunks.\n\nConstraint: CP HiCache radix and host residency are managed at page granularity, but chunked prefill can end on a sub-page tail.\nRejected: Continue skipping chunked backup | loses middle-chunk reuse and triggers fallback storms.\nRejected: Insert sub-page chunk tails into CP HiCache | creates stale-tail prune/split conflicts against request-owned KV.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce chunked_req backup skip; keep CP HiCache chunked inserts page-complete unless the radix/host cache contract becomes sub-page aware.\nTested: python -m py_compile python/sglang/srt/mem_cache/radix_cache.py python/sglang/srt/mem_cache/hiradix_cache.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py\nTested: g0034 targeted chunked CP HiCache regression tests, 2 passed\nTested: g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py, 107 passed\nNot-tested: Fresh chunked-prefill ETE traffic run after process restart
2026-06-02 06:22:17 +08:00
laoyao0822andOmX 5bd68768d9 Prevent unattached CP HiCache write acks from crashing chunked prefill
Prepared per-layer CP HiCache backups can enqueue their final write ack before the radix node is attached. Chunked prefill exposed this window when a separate catch-up backup made writing_check drain the ack queue and pop an unattached node id.

Keep ready but unattached prepared acks queued until insert either attaches the prepared backup or rollback removes the orphan ack. Also document the reactive host free-room eviction plan separately from this state-machine fix.

Constraint: CP HiCache prepared backup transfer can complete before radix insertion attaches node state

Rejected: Drop unknown ack ids | would orphan a later successful prepared attach and leak write state

Rejected: Chunked-only guard | the invalid assumption is in the generic CP write ack state machine

Confidence: high

Scope-risk: narrow

Directive: Do not drain CP write acks unless every ack id is registered in pending_host_backups or ongoing_write_through

Tested: Remote red-green test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_writing_check_defers_unattached_prepared_ack

Tested: Remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py (116 passed)

Tested: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py

Not-tested: Full chunked-prefill ETE replay after this commit

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-02 05:17:42 +08:00
laoyao0822andOmX c2d25ff591 Prevent CP HiCache pending-backup splits from killing chunked prefill
Chunked prefill can revisit a sub-page CP HiCache tail while a per-layer backup is still in flight. The old insert path split the radix node first and only then tried to prune the stale tail, so an unprunable pending backup raised after tree mutation and propagated to the scheduler.

This makes split/prune atomic from the radix-tree perspective: drain completed write acks only on the conflict path, preflight pending/unprunable state before split, and return a deferred insert result when the backup is still in flight. Unfinished requests keep their KV ownership for transfer/release instead of rematching or freeing pages under a stale tree state.

Constraint: CP HiCache backup metadata is node/page owned and cannot be repartitioned while per-layer D2H is pending
Rejected: Split the pending backup node | would require repartitioning in-flight backup metadata and host reservations
Rejected: Delete the stale tail unconditionally | risks freeing device/host pages still owned by pending backup
Confidence: high
Scope-risk: moderate
Directive: Do not move stale-tail prune after split without a preflight; pending backup split conflicts must remain non-mutating
Tested: remote g0034 container py_compile for mem_cache files
Tested: remote g0034 PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py => 115 passed
Tested: local git diff --check
Not-tested: full chunked prefill ETE after service restart
Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-02 04:51:39 +08:00
laoyao0822 d1627d1da3 Preserve CP HiCache page-aligned transfer findings
The LPF direct-kernel work exposed an allocation-dependent performance boundary: layer-page-first reduces descriptors only when host pages have contiguous extents. Capture the benchmark evidence and owner-lane allocator interpretation so future changes do not rediscover the same constraint.

Constraint: Current CP compute-owner allocation selects pages by modulo owner lane, e.g. (page_id - 1) % cp_size.

Rejected: Document LPF as an unconditional production improvement | random and owner-lane same-layout benchmarks are neutral without compact allocation.

Rejected: Treat owner_lane benchmark as full allocator replay | it models the modulo-lane constraint, while free/release history can add fragmentation.

Confidence: medium

Scope-risk: narrow

Directive: Tie any production LPF switch to compact/extent-aware host allocation or gather/staging support.

Tested: Local doc marker check for C115/C116/C117.

Tested: Remote tai-kernel CUDA pytest for the referenced kernel/test changes -> 45 passed in 2.52s.

Not-tested: Full SGLang ETE after changing host allocation policy; no such policy change is included.
2026-06-02 00:16:03 +08:00
laoyao0822 6ef4face89 Preserve FP8 CP shared-KV page contracts
NSA FP8 CP shared-KV reuse must operate on packed page-slot rows, not bf16 compact rows. The change keeps current-only and partial-current reuse inside the page-aligned materialization contract, fails fast for non-page-aligned CP split inputs, and prevents FP8 FlashMLA-KV prefill from reaching incompatible in-seq CP metadata.

Constraint: NSA FP8 persistent MLA KV rows are packed 656-byte records and CP shared KV cache management is page-granular.\nConstraint: FlashMLA-KV prefill metadata is not CP-local after NSA in-seq splitting.\nRejected: Silently splice bf16 current rows into FP8 materialized cache | corrupts the packed cache layout.\nRejected: Keep FP8 FlashMLA-KV auto prefill under NSA CP | reaches num_splits shape errors after q-row splitting.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable FP8 FlashMLA-KV prefill for NSA in-seq CP until metadata is rebuilt after CP splitting or made CP-local.\nTested: Local git diff --check and py_compile for touched SGLang files.\nTested: Remote g0034 related unit sweep recorded in docs: test_nsa_cp_utils.py, test_cp_shared_kv_layout.py, test_cp_shared_kv_runtime.py, test_cp_hicache_metadata.py passed.\nNot-tested: Full FP8 ETE startup and performance run after this commit.
2026-06-01 03:33:44 +08:00
laoyao0822 46be97adc0 Align CP shared-KV prefetch with the attention overlap window
Launching CP shared-KV prefetch from MLA prepare or the indexer can make
next-layer prefix work overlap current-layer MQA/materialization instead of
the attention window. Centralize the launch in the NSA backend after
current-layer materialization and before attention, and leave the early
indexer hook inert so the call site cannot regress silently.

The accompanying notes capture the draft-as-forward-layer follow-up plan and
the latest OOM diagnosis: the observed 178945-token failure matches the CP
in-seq MQA logits allocation, so the follow-up fix is q-dimension chunking in
_get_topk_ragged_with_cp(), not a max-prefetch-size gate.

Constraint: CP shared-KV prefetch must avoid overlapping current-layer MQA/materialization and must not add silent fallback behavior.
Rejected: Limit maximum prefetch size | hides the CP logits peak and can reduce cache/prefetch effectiveness.
Rejected: Keep prepare/indexer launch sites | they place next-layer collectives in the wrong overlap window.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce early CP prefetch launch without checking Nsight overlap and CP MQA memory peaks.
Tested: Local git diff --check and py_compile for touched Python files.
Tested: Remote container py_compile plus targeted pytest: 3 passed, 5 warnings.
Not-tested: Full ETE under production traffic after this commit.
Not-tested: CP in-seq MQA logits chunking; documented as follow-up.
2026-06-01 01:09:43 +08:00
laoyao0822andOmX 4342de0463 Avoid redundant CP collectives on sync shared-KV materialize
Synchronous CP shared KV full-hit and partial-current paths can now use the tai-kernel CUDA IPC slot-dense materialize path instead of first copying local owner pages and then running a dense CP all-reduce. This keeps the async prefetch pipeline unchanged while routing the safer synchronous runtime path through descriptor-driven owner-page reads.

The runtime builds descriptors from the existing page-aligned slot contract, caches peer pointer tables for long-lived KV/index buffers, and falls back with explicit warnings if the tai-kernel IPC capability is missing. Unit coverage locks offset handle exchange, descriptor construction, and both MLA/index full and partial-current calls.

Constraint: Async prefetch currently has unresolved scheduling contention with GEMM/MoE and current-layer KV collectives, so this commit intentionally does not wire IPC into prefetch.

Constraint: tai-kernel must provide offset-aware CUDA IPC symbols from af9fb67.

Rejected: Replace prefetch all-reduce in this slice | Nsight shows prefetch timing and communicator contention need a separate scheduling design.

Rejected: Fail-fast on missing tai IPC immediately | remote ETE still needs to validate production deployment capability before removing the warning fallback.

Confidence: medium

Scope-risk: moderate

Directive: Keep prefetch and synchronous materialize decisions separate until prefetch has a low-SM/copy-engine schedule.

Related: tai-kernel af9fb67

Tested: git diff --check; remote runtime/unit evidence recorded in docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md.

Not-tested: Fresh GLM5 ETE after this commit; async prefetch IPC path.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-01 00:00:15 +08:00
laoyao0822 b4f5c3bc0e Record CP shared KV IPC materialize evidence before runtime wiring
The materialize transport direction changed after measurement: direct CUDA IPC peer-page materialize was initially slow because of an under-parallel launch policy, not because slot-dense materialize was inherently too expensive. Capture the corrected evidence and the remaining production constraints before SGLang runtime wiring starts.

Constraint: Current near-term goal accepts SM-consuming kernels for low latency/high throughput; low-SM prefetch friendliness is deferred.

Rejected: Treat the first slow IPC result as a design blocker | tuned launch parameters beat dense all-reduce across the measured 4k-120k prefix range.

Rejected: Switch consumers to owner-concat immediately | slot-dense fused materialize is competitive enough to preserve the current consumer contract for the next integration step.

Confidence: medium

Scope-risk: narrow

Directive: Keep this document updated with every benchmark/result correction to avoid re-litigating stale conclusions.

Tested: Documentation update based on remote g0034 tai-kernel CUDA tests and cp_shared_kv_ipc_sm_tuned_20260531_181926 benchmark log

Not-tested: SGLang runtime ETE serving with IPC materialize enabled
2026-05-31 18:28:59 +08:00
laoyao0822 a149289554 Ground CP shared KV collective choices in production-shaped evidence
Add a production-style CP shared KV collective benchmark and keep the page-aligned cache contract ledger current with the collective, unpack, multi-extend prefix, P2P, and CUDA IPC findings. The benchmark makes the final consumer layout explicit so dense all-reduce and owner-packed all-gather are compared after producing the same logical-dense result.\n\nConstraint: Prefixes may be assembled from multiple radix/cache extents, so single-run zigzag ordering is only a gated fast path.\nRejected: Treat rank-major all-gather output as the final product | consumers require logical-dense ordering.\nRejected: Switch index materialization based on noisy microbenchmarks | current evidence is not strong enough for production.\nConfidence: medium\nScope-risk: narrow\nDirective: Do not replace generic logical-dense materialization with zigzag ordered gather unless a run descriptor proves the prefix shape.\nTested: python -m py_compile benchmark/hicache/bench_cp_shared_kv_production_collective.py\nTested: Remote production-style benchmark logs under /mnt/beegfs/cjy/log/cp_shared_kv_collective_bench_*_20260531_*.log\nNot-tested: Production SGLang ETE with a CUDA IPC/P2P materialization path; this commit only adds benchmark/documentation surfaces.
2026-05-31 17:04:00 +08:00
laoyao0822 71872bb851 Expose empty chat messages behind conversion failures
The replay failure returns HTTP 400 before generation, so the next diagnostic point must be the chat-template conversion boundary rather than HiCache or scheduler paths. This adds a failure-only summary that records message roles, content shapes, empty indices, tool metadata, bootstrap room, and template mode without logging raw user content.

Constraint: Production replay traffic can contain large/private messages, so diagnostics must avoid full content dumps.

Rejected: Enable full request logging | too noisy and exposes complete prompt payloads.

Rejected: Normalize empty content immediately | would hide whether the malformed message is user, assistant, tool, or template-derived.

Confidence: high

Scope-risk: narrow

Directive: Keep this log failure-only; do not move it to the hot path or print raw content.

Tested: python -m py_compile python/sglang/srt/entrypoints/openai/serving_chat.py

Tested: remote docker py_compile after scp to g0034:/mnt/beegfs/cjy/sglang-dev

Not-tested: Full replay reproduction; requires user-driven traffic after service restart.
2026-05-31 04:06:50 +08:00
laoyao0822 0fc95b6439 Keep CP HiCache reuse page-safe without eviction log churn
Page-aligned CP shared KV can pad out_cache_loc beyond valid current rows, so current reuse now gates MLA composition on the valid extend rows and permits draft partial-current reuse once the TAI sparse-page capability check passes. The TAI current-slot path self-tests sparse pages before use and falls back to the torch reference when the installed kernel is stale.

Eviction success and no-op diagnostics were also moved from INFO to DEBUG so owner-lane and host-admission churn does not flood production logs; true write failures remain WARNING.

Constraint: CP shared KV uses page-aligned physical reservations where valid suffix rows can be shorter than padded out_cache_loc.

Constraint: Production failure/fallback logs must remain visible, but hot successful eviction paths should not emit INFO per victim/rank.

Rejected: Keep draft partial-current reuse disabled | would preserve avoidable full materialization on draft cache-hit suffixes.

Rejected: Trust the TAI current-slot kernel unconditionally | stale kernels can corrupt sparse current-page composition.

Confidence: medium

Scope-risk: moderate

Directive: Do not reintroduce INFO logging in eviction hot paths without rate limiting and runtime evidence.

Tested: local py_compile for touched Python files

Tested: local git diff --check

Tested: remote container py_compile for touched Python files

Tested: remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py::TestHiCacheEvictLoggingLevels::test_evict_hot_path_success_logs_are_debug_only test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 82 passed, 5 warnings, 2 subtests passed

Not-tested: full ETE traffic after this commit; draft partial-current accept length still needs user-driven runtime validation
2026-05-31 03:31:06 +08:00
laoyao0822 3c14b1f127 Enable index partial-current reuse without replaying prefix materialize
The index path now mirrors the target MLA partial-current contract: prefetched or synchronously materialized prefix pages are composed with valid current index K/scale rows in slot-dense page buffers. Current-only batches keep the compact current-index fast path, while partial cache-hit batches share one composed dense index buffer across the in-seq prev/next topk pair.\n\nThe prefetch consume path remaps through the slot page inverse instead of treating the slot-dense buffer as physical-pool capacity, and current-index quantization uses valid extend rows so padded out_cache_loc does not disable reuse.\n\nConstraint: CP shared KV remains page-slot based; padding rows must stay invisible to attention/index semantics\nConstraint: Draft/EAGLE partial-current reuse remains guarded by should_reuse_current_extend_kv\nRejected: Replace prefix all-reduce with all-gather | NCCL all-gather still uses SM and would require an additional compose/scatter step\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce current-only gating for index reuse; partial target cache hits must compose prefix + valid current rows\nTested: Local py_compile for touched Python files\nTested: g0034 sglang-glm5-dev-2 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 77 passed, 5 warnings, 2 subtests passed\nNot-tested: Full ETE traffic with latest commit; CUDA perf impact of index partial-current prefetch under production load
2026-05-31 02:31:09 +08:00
laoyao0822 251a48fb0a Stabilize CP HiCache page-first direct transfers on CUDA 13
CP HiCache direct/page_first_direct all-layer backup was still able to enter sgl-kernel's stale cudaMemcpyBatchAsync path, which segfaults under CUDA 13 before Python can surface an error. The SGLang route now avoids that all-layer sgl-kernel op for page_first_direct backup and uses the TAI per-layer direct LF->PF op for MHA, MLA, and NSA indexer data.\n\nThe load-back path also prepares NSA indexer page indices once per load op and reuses them across per-layer H2D loads, preserving per-layer overlap while removing redundant page-index derivation.\n\nConstraint: Remote runtime is CUDA 13.0 where sgl-kernel's all-layer direct LF->PF op uses the wrong cudaMemcpyBatchAsync ABI.\nRejected: Patch sgl-kernel in this branch | we are converging production HiCache direct/page_first_direct paths onto tai-kernel and do not want to maintain another CUDA-ABI-sensitive copy path here.\nRejected: Collapse H2D load-back into one all-layer op | that would reduce submit count but lose per-layer completion visibility and forward overlap.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce sgl_kernel.transfer_kv_all_layer_direct_lf_pf for direct/page_first_direct HiCache backup without CUDA 13 ABI verification.\nTested: g0034 container: PYTHONPATH=python python -m pytest -q -s test/registered/unit/mem_cache/test_nsa_pool_host_unit.py -> 10 passed, 3 warnings.\nTested: g0034 container: PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py -> 61 passed, 3 warnings.\nTested: python -m py_compile python/sglang/srt/mem_cache/memory_pool_host.py python/sglang/srt/managers/cache_controller.py test/registered/unit/mem_cache/test_nsa_pool_host_unit.py test/registered/unit/managers/test_hicache_controller_cp.py\nNot-tested: Full ETE prefill/decode traffic after this commit.\nNot-tested: sgl-kernel implementation itself remains unchanged.
2026-05-31 01:26:07 +08:00
laoyao0822 b328baec7c Stabilize EAGLE draft cache hits under CP HiCache
The failing runs showed EAGLE accept length collapsing when draft cache-hit suffixes used the new partial-current splice path.  This keeps target partial-current reuse enabled, but returns EAGLE/NextN draft cache-hit suffixes to the previous full-materialize path with an explicit fallback warning until the draft splice path has value-level ETE proof.\n\nThe same change set also tightens the page-granular CP HiCache contract for scheduler-visible hits and makes the prefill-to-decode EAGLE handoff observable without cloning hot-path metadata.  Exact non-page CP hits are floored to a page boundary for new scheduling decisions, while internal unfinished-request refresh keeps its exact accounting.\n\nConstraint: CP shared KV and HiCache operate at page granularity; exposing token-precise CP tails to scheduler-visible cache hits can force non-page partial materialization.\nConstraint: EAGLE/NextN draft has only one executable layer, so draft prefetch and draft partial-current splice need a separate correctness contract from target layers.\nRejected: Keep draft partial-current splice enabled | remote logs correlate it with avg accept length around 0.068 and median 0.\nRejected: Clone decode metadata tensors on transfer | slot ownership until process_prebuilt consumes them avoids extra hot-path copies.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable draft partial-current reuse without metadata/draft-KV value checks and ETE accept-length evidence.\nTested: g0034 container py_compile for touched modules.\nTested: g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 183 passed, 5 warnings, 2 subtests passed.\nNot-tested: Fresh ETE accept-length run after this exact commit; requires user-driven traffic restart.
2026-05-30 22:31:43 +08:00
laoyao0822 10296a5fef Prevent stale CP HiCache tails from overlapping new page owners
CP HiCache owns KV at page granularity, but exact valid-tail extension and backed partial-tail split could leave an old sub-page tail child beside a new suffix that reuses the same physical page. That makes radix residency ambiguous across device, host, and draft mirrors. The insert/match split paths now prune stale floored tails when safe, and defer/fail through the existing pending-split path when the subtree is protected or has in-flight backup state.\n\nThis also keeps a temporary scheduler boundary warning for externally observed zero-output responses so future ETE runs can classify whether zero visible output reaches SGLang's output processor.\n\nConstraint: CP shared KV and HiCache manage physical KV by page, while radix keys retain valid-token lengths.\nRejected: Keep overlapping old tail nodes after page-floor split | leaves two independent cache states for one physical tail page.\nRejected: Force-prune protected or in-flight backup tails | can mutate cache state still used by active transfer or inference.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not remove the stale-tail prune without replacing it with another page-granular ownership rule for CP HiCache radix splits.\nTested: Remote py_compile for hiradix_cache.py and scheduler_output_processor_mixin.py in g0034 container.\nTested: Remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py -> 97 passed, 5 warnings.\nNot-tested: Full ETE recovery of EAGLE accept length; latest ETE still shows accept collapse, documented in C55.\nNot-tested: Router/client-side output_len=0 correlation when server-side OUTPUT_ZERO_DEBUG does not fire.
2026-05-30 05:51:53 +08:00
laoyao0822 e9c341afe8 Hold decode EAGLE metadata until prebuilt consumption
Transferred EAGLE metadata buffers are reusable slot views. The previous clone-based mitigation protected correctness but added copies on the transfer hot path and hid the actual lifetime contract. This change makes the transferred request own the metadata slot while it sits in the decode waiting queue, then releases it immediately after process_prebuilt has consumed top-k and hidden state into the prebuilt batch. Abort paths also release any held decode metadata slot.

Constraint: Decode disaggregation metadata buffers are reusable slot views consumed later by process_prebuilt.

Rejected: Clone transferred EAGLE tensors at commit time | correct but less efficient and masks the ownership contract.

Rejected: Release in process_batch_result_prebuilt | holds slots across forward longer than needed.

Confidence: medium

Scope-risk: moderate

Directive: Do not free successful EAGLE transfer metadata in pop_transferred unless process_prebuilt consumption is also moved earlier.

Tested: Remote py_compile for decode.py, scheduler.py, scheduler_output_processor_mixin.py, and test_decode_queue_compaction.py.

Tested: Remote focused lifecycle tests passed: 3 passed.

Tested: Remote full test_decode_queue_compaction.py passed: 11 passed, 5 warnings.

Not-tested: Fresh ETE runtime validation of EAGLE accept-length recovery after the C48 sync.
2026-05-30 04:51:50 +08:00
laoyao0822 07c9544737 Preserve transferred EAGLE state past metadata-slot reuse
Decode committed EAGLE top-k and hidden-state tensors as views into reusable metadata-buffer rows. The metadata index is freed immediately after transfer commit, while the request may wait before process_prebuilt consumes the draft state. Under concurrent cache-hit traffic a later transfer can overwrite the same row, leaving output_id copied correctly but EAGLE draft state corrupted, which matches low accept length despite successful KV/state registration.

Constraint: Metadata slots are intentionally recycled right after transfer commit for throughput.

Rejected: Hold metadata slots until process_prebuilt | larger lifetime change and reduces transfer capacity; cloning the small prebuilt EAGLE state is narrower.

Confidence: high

Scope-risk: narrow

Directive: Do not store reusable metadata-buffer views on Req unless the slot lifetime is extended through all consumers.

Tested: Local py_compile for decode.py and test_decode_queue_compaction.py.

Tested: Remote g0034 container py_compile for decode.py and test_decode_queue_compaction.py.

Tested: Remote g0034 focused clone-lifetime test: 1 passed.

Tested: Remote g0034 test_decode_queue_compaction.py: 10 passed, 5 warnings.

Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
2026-05-30 03:36:42 +08:00
laoyao0822 b7364d23f9 Keep draft cache-hit KV on current-suffix compose
Cache-hit EAGLE/NextN draft was falling back to full materialization while the target path used page-aligned prefix materialize plus fresh current-suffix splice. That creates a target/draft KV source asymmetry exactly on the high-cache-hit path where decode accept length collapsed. The draft model still does not get next-layer async prefetch; only the same-layer current suffix compose contract is made role-agnostic.

Constraint: EAGLE/NextN has no real next decoder layer, so draft async prefetch remains disabled.

Rejected: Restore draft full-materialize fallback | recreates the observed target/draft cache-hit asymmetry and hides stale-current-suffix bugs.

Confidence: medium

Scope-risk: moderate

Directive: Do not reintroduce draft cache-hit current-reuse fallback without proving the draft persistent pool has fully fresh suffix rows before attention.

Tested: Local py_compile for cp_shared_kv_runtime.py and test_cp_shared_kv_runtime.py.

Tested: Remote g0034 container py_compile for changed runtime/test files.

Tested: Remote g0034 pytest test_cp_shared_kv_runtime.py: 73 passed, 5 warnings, 2 subtests passed.

Tested: Remote g0034 related suite test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py: 128 passed, 5 warnings, 2 subtests passed.

Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
2026-05-30 03:06:17 +08:00
laoyao0822 5f343c65ca Keep EAGLE diagnostics from breaking disagg startup
The C32 startup audit was diagnostic-only, but it incorrectly read spec_algorithm from helper queue objects. PrefillBootstrapQueue and DecodePreallocQueue keep scheduler-owned runtime state instead, so the debug path failed before KV manager initialization could complete.\n\nUse scheduler.spec_algorithm for both startup audits and record the ownership rule in the page-aligned cache ledger.\n\nConstraint: Queue helpers do not copy every scheduler field onto self.\nRejected: Disable EAGLE accept diagnostics | would lose the draft-transfer evidence needed for the accept-len investigation.\nConfidence: high\nScope-risk: narrow\nDirective: Disaggregation helper diagnostics should read scheduler-owned runtime state unless the constructor explicitly copied the field.\nTested: Local py_compile for prefill.py and decode.py.\nTested: Local git diff --check.\nTested: Synced prefill.py/decode.py to g0034 and remote container py_compile passed.\nNot-tested: Full disaggregated startup and ETE traffic after this fix; user controls runtime launch and load generation.
2026-05-30 01:37:05 +08:00
laoyao0822 b56a4f2e6b Stabilize CP HiCache page-tail ownership under EAGLE reuse
CP shared KV and HiCache now keep page-aligned physical ownership while preserving valid-token radix semantics. Repeated tiny EAGLE exact hits free duplicate tail pages instead of leaking one allocator page, owner-lane load-back uses page-vector admission/eviction, and single-DP idle schedulers avoid entering an unnecessary MLP-sync collective.

The commit also records the current page-aligned cache contract and adds gated decode-side EAGLE accept diagnostics so future accept-length collapses can be tied to draft KV/state transfer evidence instead of more prefill cache speculation.

Constraint: CP HiCache allocator ownership is page-granular while radix matching remains valid-token based.

Constraint: New diagnostics must be gated and must not alter normal EAGLE, transfer, or cache behavior.

Rejected: Padding short requests to cp_size or 2*cp_size pages | wastes KV capacity and still hides valid-tail lifecycle bugs.

Rejected: Adding more unconditional collectives to prove CP consistency | hot-path collectives previously caused severe performance risk.

Confidence: medium

Scope-risk: broad

Directive: Do not reintroduce silent fallback for CP shared KV/HiCache paths; warning-level fallback or fail-fast is intentional.

Tested: git diff --check

Tested: local py_compile for all modified Python files

Tested: remote g0034 container py_compile for modified Python/test files

Tested: remote g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/managers/test_scheduler_dp_attn_mixin.py => 114 passed, 5 warnings, 2 subtests passed

Not-tested: full ETE traffic rerun after this commit

Not-tested: CUDA/TAI kernel benchmark coverage for all production shapes
2026-05-30 01:20:01 +08:00
laoyao0822 21065cdfdf Keep tiny CP cache-hit suffixes off async prefetch
The repeated-request hang reproduced on g0034 after the cached-prefix
request created both MLA and index async prefetchers with
prefix_lens=[40320] and extend_lens=[65]. The existing one-page default
only blocked sub-page suffixes, so a barely-over-one-page suffix still
entered the next-layer collective path before any later forward progress
was logged.

Raise the default async prefetch extend gate to one page per CP lane while
keeping the env override. This only gates async prefetcher object
creation; target partial-current reuse still uses the synchronous
page-slot compose/current-splice path when no prefetcher exists.

Constraint: cp_size=8,page_size=64 repeated prompt had extend_len=65 and hung immediately after has_mla=True has_index=True create_result logs
Rejected: Disable partial-current reuse for short extends | that would lose the cache-hit benefit and regress current/full reuse
Rejected: Disable all async prefetch by default | broader performance impact than the observed tiny-suffix failure
Confidence: medium
Scope-risk: moderate
Directive: Do not lower the default below one page per CP lane without ETE proof that repeated cache-hit tiny suffixes no longer hang
Tested: Remote g0034 container py_compile for touched runtime/prefetch/test files; targeted C22 tests passed 2 tests plus 2 subtests; full test_cp_shared_kv_runtime.py passed 73 tests plus 2 subtests
Not-tested: Full multi-node ETE repeated-request run after this threshold change
2026-05-29 22:04:23 +08:00
laoyao0822 40cf691c78 Keep CP HiCache valid tails page-owned during extension
CP shared KV HiCache now treats non-page-aligned valid-tail nodes as page-owned when a later request extends beyond them. The prefix probe, match path, insert path, and prepared backup start now agree on flooring the reusable prefix to the previous physical page boundary, so prepared backup metadata cannot start mid-page or fail to attach at insertion.

Duplicate frees under CP HiCache now go through a page-safe free helper. Insert and unfinished duplicate ranges free only fully unprotected pages; no-insert completion still releases the right tail owned only by the finishing request.

Constraint: CP HiCache allocator frees whole physical pages even when called with token-granular locs.

Rejected: Partial-page sharing/refcounting | too complex for the current page-as-minimum-unit contract.

Rejected: Fix only prepare_write_backup_for_req | match_prefix and insert would still expose exact valid-tail hits and desynchronize prepared backup length.

Confidence: medium

Scope-risk: moderate

Directive: Do not expose non-page-aligned CP valid-tail hits to extending requests unless partial-page ownership is explicitly implemented end-to-end.

Tested: remote g0034 py_compile for touched files

Tested: remote g0034 test_cp_hicache_metadata.py 97 passed

Tested: remote g0034 test_cp_shared_kv_runtime.py 73 passed

Not-tested: test_cp_shared_kv_layout.py aborts during installed sgl_kernel architecture-specific op loading before assertions
2026-05-29 21:49:17 +08:00
laoyao0822 2a9dfcca6f Keep current reuse independent of async CP prefetch
Async MLA/index prefetch is a scheduling optimization, not the correctness contract for target current reuse. Tiny cache-hit suffixes can skip async prefetcher creation while target partial-current reuse still composes page-slot prefix materialization with current KV rows synchronously. CP HiCache radix/device accounting now treats retained valid-tail pages as physical page spans so allocator state stays consistent when logical cache keys are shorter than the retained page.

Constraint: CP shared KV ownership and HiCache residency are page-granular while request-visible cache lengths remain valid-token lengths.
Constraint: Async prefetch can hang or regress on large-prefix tiny-extend traffic and must not be required for current reuse.
Rejected: Treat missing prefetcher as fail-fast for target partial-current reuse | disabled useful current reuse and broke tiny-prefix/tiny-suffix traffic.
Rejected: Keep async prefetcher object with synchronous consume mode | conflates prefetch object existence with current-layer correctness and hides fallback semantics.
Confidence: medium
Scope-risk: moderate
Directive: Do not make current-only or target partial-current reuse depend on MLA/index prefetcher creation; prefetcher objects mean async next-layer work exists.
Tested: Remote g0034 container py_compile for touched modules.
Tested: Remote g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 73 passed.
Tested: Remote g0034 PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py -> 90 passed.
Not-tested: Latest full ETE traffic run with GLM-5.1 CP HiCache after this commit.
Not-tested: CUDA kernel-level performance impact of synchronous no-prefetch partial-current compose.
2026-05-29 19:34:27 +08:00
laoyao0822andOmX 2b1524bd8c Apply CP HiCache page-floor policy before backup and insert
Backed CP HiCache tail splits were floored during match, but write preparation and radix insertion could still observe the raw sub-page prefix. That could skip one token during pre-forward reservation or call CpHiCacheNodeMetadata.split() with a non-page boundary. The shared helper now floors backed partial-tail prefixes before probe, match, and insert.

Constraint: CP HiCache uses page-granular physical ownership while radix keys may keep valid-tail lengths.

Rejected: Let probe and insert keep token-precise tail prefixes | it reintroduces half-page ownership through a different entry point.

Confidence: high

Scope-risk: moderate

Directive: Any CP radix path that can split or reserve against backed HiCache metadata must use the same page-floor policy.

Tested: Remote red tests first for probe and insert failures in g0034 container.

Tested: Remote py_compile for hiradix_cache.py and test_cp_hicache_metadata.py.

Tested: Remote targeted C7 tests: 4 passed, 3 warnings.

Tested: Remote pytest test_cp_hicache_metadata.py test_cp_hicache_load_back_owner_lanes.py: 94 passed, 5 warnings.

Not-tested: Live ETE traffic with concurrent divergent tail prompts.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-05-29 07:10:53 +08:00
laoyao0822andOmX 8001c4ae8e Floor CP HiCache tail splits to page boundaries
CP HiCache ownership is page-granular, so a backed radix node must not be split inside a padded physical tail page. When a shorter hit would require an interior tail-page boundary, matching now floors to the previous page boundary and sacrifices the sub-page cache prefix instead of splitting ownership metadata.

Constraint: Host/device/draft CP HiCache metadata tracks page owners and padded physical spans.

Rejected: Split one padded tail page across two radix nodes | it would require half-page ownership semantics and risks double-counting capacity.

Confidence: high

Scope-risk: moderate

Directive: Keep exact valid-tail hits, but floor partial backed-node splits to page boundaries unless metadata gains explicit sub-page ownership.

Tested: Remote py_compile for hiradix_cache.py and test_cp_hicache_metadata.py in g0034 container.

Tested: Remote pytest targeted backed-tail split tests plus exact valid-tail hit test: 3 passed.

Tested: Remote pytest test_cp_hicache_metadata.py test_cp_hicache_load_back_owner_lanes.py: 92 passed, 5 warnings.

Not-tested: Live ETE traffic under divergent short-prefix prompts.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-05-29 06:57:16 +08:00
laoyao0822andOmX c9f790cde9 Fail fast when CP MLA partial-current compose is unavailable
Mixed prefix/current MLA reuse depends on the page-slot prefetch compose path to preserve padded tail semantics. The compact materialize/current merge path can re-expose suffix slack as valid dense rows, so the backend now raises with an explicit fail-fast marker instead of silently falling back.

Constraint: Page-aligned CP shared KV contract requires suffix tail slack to remain invalid.

Rejected: Keep compact merge as a fallback | it has a different dense-row contract and can hide correctness bugs.

Confidence: high

Scope-risk: moderate

Directive: Do not reintroduce merge_materialized_and_current_kv on MLA partial-current reuse without a page-slot correctness proof.

Tested: Remote py_compile for nsa_backend.py and test_nsa_cp_utils.py in g0034 container.

Tested: Remote pytest test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_mla_partial_current_path_fails_fast_instead_of_compact_fallback.

Tested: Remote pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py: 124 passed, 5 warnings.

Not-tested: Live ETE traffic and CUDA kernel execution for this fail-fast path.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-05-29 06:46:39 +08:00
laoyao0822andOmX 4f9bb7ce30 Record CP shared-KV C11-C14 closure evidence
C11-C14 are now tied to explicit remote verification instead of leaving the ledger at the intermediate failure state.  The mixed-version symbol issue is documented as a caller-sync problem, and the stale C14 expectations are recorded as intentional current contracts rather than open regressions.

Constraint: Local pytest in this workspace is blocked by missing optional runtime dependencies such as orjson, so remote g0034 container tests are the authoritative CPU/unit evidence for these files.

Rejected: Reintroduce cp_shared_kv_mla_prefetch_should_trace_tiny_extend | the current callers no longer import it, and adding a dead compatibility helper would hide sync mistakes.

Confidence: high

Scope-risk: narrow

Directive: Keep C13 validation focused on matching caller/runtime sync; do not resurrect stale helper APIs unless an active caller contract requires them.

Tested: Local py_compile for touched runtime/prefetch/caller/test files; remote caller import check printed CALLER_IMPORT_OK; remote exact C14 set passed 6 passed, 3 warnings; remote three-file suite passed 123 passed, 5 warnings.

Not-tested: CUDA/TAI ETE runtime logs were not rerun in this documentation-only commit.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-05-29 06:26:49 +08:00
laoyao0822andOmX dd69c5970c Standardize CP shared-KV runtime fallback markers
Current reuse, TAI materialize, and TAI fused MLA store are optimized CP shared-KV paths. Their fallback helpers already emitted warnings, but the messages were not consistently grep-able with the standard CP shared-KV fallback marker.

This moves the marker into the helper layer so individual fallback call sites do not have to remember to include it, and keeps the existing per-reason rate limiting.

Constraint: Runtime fallback logs must be visible without enabling debug logging.

Rejected: Prefix only selected call sites | helper-level prefixing avoids future unmarked fallback messages.

Confidence: high

Scope-risk: narrow

Directive: New CP shared-KV runtime fallback helpers should use [CP_SHARED_KV_FALLBACK] and a component name.

Tested: Local py_compile for cp_shared_kv_runtime.py and test_cp_shared_kv_runtime.py.

Tested: Local extracted assertLogs check for all three fallback helper prefixes.

Tested: Remote g0034 pytest exact runtime helper tests: 2 passed, 3 warnings.

Not-tested: Full cp_shared_kv_runtime.py suite in this slice.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-05-29 06:02:13 +08:00
laoyao0822andOmX e09c8256d7 Expose CP shared-KV allocation fallback as a warning
The compute-owner allocation path is an intended hot path for CP shared KV. Falling back to legacy page allocation changes the behavior profile enough that INFO/no-marker logging is too easy to miss during profiling.

This makes the fallback warning-visible and gives it the standard CP shared-KV fallback marker while preserving the existing behavior of logging every fallback event.

Constraint: Production fallbacks from intended CP shared-KV hot paths must be visible and grep-able.

Rejected: Keep INFO logging | historical silent or low-visibility fallbacks hid inactive optimized paths.

Confidence: medium

Scope-risk: narrow

Directive: Keep fallback logs warning-level unless the path is proven to be expected steady state and separately observable.

Tested: Local py_compile for common.py and test_cp_shared_kv_layout.py.

Tested: Local extracted assertLogs check for _log_cp_shared_kv_alloc_fallback warning prefix and reason formatting.

Tested: Remote g0034 py_compile for common.py and test_cp_shared_kv_layout.py.

Not-tested: Full remote pytest for test_cp_shared_kv_layout.py; current g0034 container aborts while importing installed sgl_kernel common_ops before test execution.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-05-29 05:59:02 +08:00
laoyao0822andOmX 38532a1fc9 Prevent incompatible CP shared-KV transfer mapping
Mooncake is the only disaggregation transfer backend in this branch with CP shared-KV owner filtering plus logical-page-position destination selection. NIXL still slices destination pages by the original chunk slice, so allowing CP shared-KV prefill on NIXL can silently pair filtered prefill pages with the wrong decode pages.

This keeps the supported path narrow while preserving the page-aligned transfer contract: non-page-aligned valid tails transfer their physical tail page, but do not get padded to CP-size pages.

Constraint: CP shared-KV transfer remaps prefill logical pages to per-rank physical pages while decode metadata remains request-position based.

Rejected: Let NIXL continue through the generic slice path | it lacks logical-page-position selection and can silently corrupt CP shared-KV transfers.

Confidence: high

Scope-risk: narrow

Directive: Do not enable CP shared-KV on another PD transfer backend until its sender filters owner pages and selects decode pages by logical request-page position.

Tested: Local py_compile for server_args and touched tests.

Tested: Remote g0034 pytest test_cp_shared_kv_transfer_mapping.py test_req_to_token_pool.py TestHiCacheArgs: 22 passed, 8 subtests passed.

Not-tested: End-to-end PD transfer with a live non-page-aligned prompt.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-05-29 05:53:37 +08:00
laoyao0822andOmX 7e06eaebbf Prove draft HiCache mirrors target valid-tail pages
Draft HiCache already piggybacks target CP reservations, but the controller coverage only exercised page-aligned nodes. The new tests pin the intended contract for valid-tail nodes: target and draft reserve/load the same padded physical page while scheduler-visible load results keep only valid logical locs.

Constraint: Draft KV is a target mirror; it must not choose an independent valid-only host path.
Rejected: Re-enable draft partial-current reuse in this slice | current safe contract keeps EAGLE/NextN cache-hit draft on full materialization until same-layer padded visibility is proven.
Confidence: medium
Scope-risk: narrow
Directive: Keep draft and target host metadata coupled; do not add draft-only capacity or prefetch decisions.
Tested: local py_compile for test_hicache_controller_cp.py.
Tested: remote g0034 test_hicache_controller_cp.py: 59 passed, 3 warnings.
Not-tested: CUDA E2E runtime for this commit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-05-29 05:43:49 +08:00