Replaces the last remaining NCCL collective in the bs>1 compose layer loop
(the compact-current all-reduce) with a DeepEP-style counting barrier plus
a CUDA-IPC gather from peers' symmetric dense buffers, behind
SGLANG_CP_SHARED_KV_COMPOSE_SYMM (+ COMPOSE_ARENA, both default off).
- CpComposeArena.register_symm: fixes capacity at the pool-derived bound
(logical pages x dense page unit, overridable via
SGLANG_CP_SHARED_KV_SYMM_HEAP_MB), allocates slab + flags in CUDA-IPC
memory, exchanges handles once over the CP group; deterministic bump
carve means peer_base + my_offset addresses any peer's dense buffer with
no per-layer handshakes. Registration happens only in the token-KV
compose (uniform first-use point); growth after registration raises.
- Current pages are single-writer at page granularity under the
page-aligned in-seq split, so the exchange is the existing
gather_cuda_ipc_peer_pages with src==dst page ids and writer (compute
owner) ranks; writers are built per batch by
build_batch_current_page_writer_ranks and gated by
maybe_build_current_page_writer_ranks (page_aligned metadata required).
The barrier runs even with zero remote pages (counts must match).
- _agreed_tai_ipc_peer_ptrs: the per-rank IPC capability probe is now
agreed across the CP group (one-time MIN all-reduce per pool tensor) so
ranks can never split between gather and collective paths and deadlock
on mismatched NCCL shapes.
- ComposePlan cache re-anchored ON the slot_remap object (forward-batch
lifetime) instead of a module-level tensor-identity key, which could go
stale when a freed tensor's address is reused by the next batch.
Validation (g0034 syh-dev-new): tai-kernel cp_symm_barrier correctness
(200 adversarial iterations, rotating 10ms producer delays, phase-safety,
flags drained at quiescence) and perf (7.7us max-rank latency) both pass;
8-rank GPU test extends to the symm path - byte-identical to compose_v2
across 4 layers (parity halves exercised, slab registered); mem_cache
suite 464 passed with the only failures being a documented pre-existing
sys.modules stub pollution pair, reproduced identically with
SGLANG_CP_SHARED_KV_COMPOSE_V2=0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bs>1 partial-current materialize issued one sum-all-reduce per request
span per buffer per layer (48 collectives per F-layer at bs=12, 2880 per
batch, 419ms + 49ms launch gaps in the production trace), all inline on
the compute stream. The data is a partitioned gather, not a reduction:
every byte has exactly one producer.
compose_v2 (SGLANG_CP_SHARED_KV_COMPOSE_V2, default on) replaces this with:
- fast path: one tai-kernel CUDA-IPC slot-dense gather covering ALL prefix
spans (full-range descriptors, -1 sentinels zero-fill current slots and
replace the dense zero-fill) + ONE collective over the compact current
pages (uint8 byte view; exact because every byte is writer-exclusive).
- fallback (no peer IPC): local materialize of all prefix spans + ONE
whole-buffer sum-all-reduce (rows are still writer-exclusive pre-reduce).
IPC capability is decided once by the cached peer-pointer probe; after a
successful probe a failing gather raises (no per-call try/except).
cp_shared_kv_compose.py adds the per-batch ComposePlan descriptor cache
(layer-invariant, keyed on the slot_logical_pages identity) and the
CpComposeArena with tier-S carve discipline (deterministic bump, layer-
parity halves; default off) so the Step B symmetric-memory conversion is
a registration flip.
Microbenchmark (g0034 8xH200, traced 12-req batch, per batch): per-span
214ms -> fused AR 119ms -> IPC prefix + compact current 84ms; symm target
61ms. Validation: 143 unit tests incl. v2-contract twins, legacy siblings
and rank-merged simulations under both paths; mem_cache dir 432 passed;
8-rank GPU byte-exactness vs legacy with real NCCL + IPC (test/manual/
test_cp_shared_kv_compose_v2_8rank.py) passed with no fallback markers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reserve_write_cp undrained-ack refusal returned a bare
HiCacheWriteFailure, which _reserve_write_cp_indices_no_collective
interpreted as a host-capacity failure: with free host space the
retry path tripped the predicted-no-deficit RuntimeError (crashing
the scheduler in exactly the scenario the gate exists to handle
gracefully), and with a deficit it triggered pointless evictions
before refusing again.
Give HiCacheWriteFailure an explicit reason (default host_capacity
keeps all existing constructors/semantics); the wrapper returns
non-capacity failures immediately as skip-this-round, and a
wrapper-level test pins that undrained_ack reaches neither the retry
admission nor host eviction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
node_has_undrained_write_ack scanned ack_write_queue on every
reservation, including the per-request-per-chunk prepare path whose
node ids are freshly minted and can never be queued. Track the max
node id ever appended to the queue: any queued id is <= max by
construction (no monotonicity assumption needed for correctness), so
fresh ids exit on one integer compare and the scan remains only for
re-reservations of old ids (the rare write_backup fallback).
All three ack_write_queue append sites now go through a single
_append_write_ack funnel that maintains the max — the zero-owned-rank
and non-CP write acks previously would have bypassed it, allowing
false negatives on ranks owning no pages of a node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dfa168abe9 made duplicate ack completion idempotent at the consumer
(writing_check), which is correct and lock-balance-safe but masks the
producer invariant breach: two acks can only coexist for one radix
registration when an ack is orphaned in ack_write_queue after a
rollback cleared ongoing_write_through/pending_host_backups,
re-opening the _node_host_write_pending guard for a fresh
registration. _rollback_pending_backup (write_backup's exception
path) was the one rollback that did not scrub the node's acks — and
it also left a half-submitted layer-write state alive, which later
forwards would keep driving, writing D2H into host slots the rollback
had already evicted.
Close the invariant structurally:
- reserve_write_cp refuses (HiCacheWriteFailure, the existing
skip-this-round path both callers already handle) when the node
still has an undrained final ack, computed by scanning the small
ack queue — no new state to keep in sync.
- _rollback_pending_backup now cancels the pending layer-write state
(write-stream sync so in-flight per-layer copies finish before the
host slots are evicted) and scrubs the node's queued acks via the
scrub factored out of _rollback_prepared_cp_backup.
- The consumer-side duplicate guard from dfa168abe9 is kept as
defense in depth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CP per-layer write-through can leave duplicated ready ack ids in the
ack queue while pending-split insertion drains write visibility. The
pre-scan observed both duplicates as registered before the first
completion removed radix state, so the second completion raised KeyError
and killed the scheduler.
The write-check path now records completed ack ids until matching queue
entries are drained. Duplicate completed acks are removed with a warning,
while genuinely unknown or unregistered acks still fail fast.
Constraint: CP HiCache pending-split drain may call writing_check outside the normal idle event path.
Rejected: Blind pop(..., None) for all unknown acks | would silently hide truly unattached or corrupt ack state.
Confidence: high
Scope-risk: moderate
Directive: Do not remove the completed-ack short-term memory unless duplicate and stale ack queues are proven impossible across TP MIN synchronization.
Tested: Local py_compile for hiradix_cache.py and test_cp_hicache_metadata.py.
Tested: Remote cjy-glm5-new py_compile and full test_cp_hicache_metadata.py, 121 passed.
Not-tested: Full ETE prefill workload after restarting service.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Over-long inputs produced two different client errors depending on
which bound rejected them: the TokenizerManager pre-check (raw
context_len) returned 413 PayloadTooLargeError ('The input (N tokens)
is longer than the model's context length (M tokens).'), while inputs
between that and the scheduler's stricter effective limit hit
validate_input_length and returned 400 BAD_REQUEST with different
wording (and a confusing 'X exceeds X' message since the check is >=).
Unify on the 413 format end to end:
- validate_input_length wording now matches the TokenizerManager
message, reporting the effective per-request limit.
- set_finish_with_abort takes status_code/err_type; the scheduler
length-rejection sites abort with REQUEST_ENTITY_TOO_LARGE +
PayloadTooLargeError. The batch handler previously queued the
over-long request WITHOUT marking it aborted (it proceeded to
prefill) — also fixed.
- Non-streaming aborts with 413 raise PayloadTooLargeError (now a
ValueError subclass so raw /generate-style endpoints that only
catch ValueError still respond; the OpenAI layer's except clause
is reordered to win and emit the 413 format).
- Streaming abort responses prefer the scheduler-provided err_type
over the HTTPStatus name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'[CP_PER_LAYER_TRANSFER] registered room=... chunk=...' fired at INFO
once per room per chunk on every prefill send. The one-time manager
registration stays INFO and failure paths stay WARNING; the
success-side finish log was already debug.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Absorb PR 11's final Para compatibility surface as an opt-in OpenAI serving layer rather than hard-coding business defaults into protocol models. The change adds server args for Para chat defaults, Kimi/GLM compatibility, tool-choice normalization, tool-role text flattening, and streaming first-chunk error preflight while preserving default upstream behavior unless explicitly enabled.
Reasoning token usage is also propagated through chat/completion usage paths, with GLM compatibility emitting completion_tokens_details.reasoning_tokens. Low-risk protocol fixes accept string image_url content parts and preserve GLM function-call argument value whitespace.
Constraint: Online Para-compatible deployments require request/response semantics that differ from default OpenAI serving behavior.
Constraint: Current CP/HiCache/bs>1 work must not be coupled to OpenAI serving compatibility changes.
Rejected: Merge PR 11 history directly | intermediate commits briefly hard-code chat max_tokens=32768 before later gating it by server args.
Rejected: Enable Para compatibility by default | would change non-Para OpenAI-compatible deployments.
Confidence: high
Scope-risk: moderate
Directive: Keep Para-specific serving policies behind explicit server args unless the business contract changes globally.
Tested: PYTHONPATH=python:. python -m unittest discover -s test/registered/unit/entrypoints/openai -p 'test_para_serving_protocol.py' -v (19 tests OK)
Tested: python -m py_compile modified OpenAI serving, tokenizer manager, server_args, function-call detector, and test files
Not-tested: Live router/prefill/decode OpenAI serving E2E after enabling Para flags.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Per-layer prefill-to-decode transfer now finishes per request/rank/chunk, so success-path INFO logs can dominate production logs and hide actual failures. Keep successful finish breakdown and completion messages at DEBUG while preserving nonzero finish status as WARNING.
Constraint: Per-layer transfer is a hot path under CP shared-KV and may produce many batch completions per request.
Rejected: Disable CP per-layer transfer logging entirely | failures still need visible warning-level evidence.
Confidence: high
Scope-risk: narrow
Directive: Do not promote successful per-request transfer completion logs back to INFO without rate limiting.
Tested: PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_cp_per_layer_transfer.py::TestPerLayerTransferContext::test_successful_finish_does_not_emit_hot_path_info_log
Tested: python -m py_compile python/sglang/srt/disaggregation/cp_per_layer_transfer.py python/sglang/srt/disaggregation/mooncake/conn.py
Not-tested: Full local disaggregation suite blocked by missing local orjson dependency.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV bs>1 cache-hit loads already merge request load ops, but the host pool still rebuilt layer-invariant mapping work from the same host/device indices. Introduce a PreparedLoadDescriptor lifecycle around begin/end load, wire MLA KV and NSA index H2D loads through tai-kernel prepared submit when available, and add timing hooks plus regression coverage for descriptor reuse and explicit fallback logging. Record the P4/P6b design and benchmark results in the advanced feature notes.
Constraint: Radix residency and allocator decisions remain synchronous; only the data-transfer descriptor is prepared for per-layer async submit.
Constraint: Production fast path must not silently fall back when tai prepared H2D support is missing.
Rejected: Cross-batch descriptor reuse | descriptor lifetime and tensor ownership are only safe within one load operation.
Rejected: Change L2->L1 scheduling to layer-ahead prefetch in this commit | that is a separate lifecycle change after descriptor reuse is stable.
Confidence: medium
Scope-risk: moderate
Directive: Keep LayerDoneCounter per-layer readiness semantics; do not replace with all-layer waits.
Tested: python -m py_compile python/sglang/srt/mem_cache/memory_pool_host.py python/sglang/srt/managers/cache_controller.py
Tested: Remote g0034:cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py (88 passed)
Tested: Remote tai-kernel prepared descriptor CUDA test (6 passed) and P4 benchmark full matrix (90 rows)
Not-tested: ETE replay/GSM8K cache-hit correctness after this commit
Not-tested: Layer-ahead L2->L1 prefetch scheduling
Co-authored-by: OmX <omx@oh-my-codex.dev>
When chunked prefill is active, CP shared-KV bs>1 cannot consume more extend
tokens than the current chunk budget. If the CP-specific extend-token limit is
omitted, default it to rem_chunk_tokens so scheduler admission reflects the
reachable chunk capacity. The request-count and cached-token knobs keep their
None-as-unlimited behavior.
Constraint: CP bs>1 batching must not advertise a larger extend batch than chunked prefill can execute.
Rejected: Require users to always set --cp-shared-kv-prefill-max-total-extend-tokens | the safe default is already available from chunked prefill state.
Rejected: Default batch request or cached-token limits | those are policy knobs and None should remain unlimited.
Confidence: high
Scope-risk: narrow
Directive: Keep --cp-shared-kv-prefill-max-total-extend-tokens as min(user_limit, chunk_budget) when both exist.
Tested: Local py_compile for schedule_policy.py and test_prefill_adder.py.
Tested: Remote g0034 cjy-glm5-new targeted prefill_adder tests: 2 passed.
Not-tested: Full ETE scheduler batching distribution after defaulting the extend limit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV batching previously estimated MQA logits from full request
extend/context rows, which overstated memory because CP in-seq split only
computes each rank's two zigzag segments. Add CP-size aware row accounting
that mirrors the fused CP MQA materialization path and take the worst local
rank peak for scheduler admission.
Expose SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_GB as a more direct cap for one fp32
MQA logits chunk. Runtime and scheduler now both translate this GB cap into
chunk rows from the actual K rows, while keeping the old row cap as a
mutually-exclusive expert override.
Constraint: Scheduler admission must stay CUDA-sync-free and use static budget information only.
Rejected: Keep full-request q*k admission | it over-gates CP bs>1 batches because CP splits q rows per rank.
Rejected: Let rows and GB caps both apply | precedence would be ambiguous during tuning.
Confidence: medium
Scope-risk: moderate
Directive: Keep MQA logits admission tied to the fused CP MQA segment shape; do not revert to full request token counts.
Tested: Local py_compile for touched runtime, scheduler, estimator, and tests.
Tested: Local pytest test_cp_shared_kv_prefill_buffer_estimator.py: 8 passed.
Tested: Remote g0034 cjy-glm5-new py_compile and targeted estimator/runtime tests: 13 passed.
Not-tested: Full ETE high-cache-hit CP bs>1 load with SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_GB.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV bs>1 admission already bounds request count, extend tokens,
cached tokens, and an estimated temporary buffer size. The estimate missed
the fp32 MQA logits temporary, whose peak grows with query rows times
context rows and can dominate high-cache-hit multi-request batches.
Add an MQA logits peak term to the CPU-only estimator and include it in
the layer-forward peak enforced by --cp-shared-kv-prefill-max-buffer-size.
When SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_ROWS is set, admission estimates the
post-chunk peak using that row cap; otherwise it remains conservative and
assumes the full extend-row count.
Constraint: Scheduler admission must stay CPU-only and cannot query CUDA free memory.
Rejected: Add a separate scheduler limit for MQA logits | the existing max-buffer-size knob is the right aggregate admission budget.
Rejected: Use SGLANG_NSA_MQA_LOGITS_FREE_MEM_FRACTION in scheduler | that depends on runtime CUDA free memory and would make admission host-sync or stale.
Confidence: medium
Scope-risk: moderate
Directive: Keep the estimator conservative when chunk max rows is unset; do not rely on CUDA free-memory queries in scheduler admission.
Tested: Local py_compile for estimator, scheduler, schedule_policy, and estimator tests.
Tested: Local pytest test_cp_shared_kv_prefill_buffer_estimator.py: 5 passed.
Tested: Remote g0034 cjy-glm5-new py_compile and estimator pytest: 5 passed.
Not-tested: ETE scheduler admission under high-cache-hit bs>1 traffic.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV bs>1 can build large fp32 MQA-logits temporaries from
DeepGEMM fp8_mqa_logits. The official SGLang path already chunks normal NSA
MQA logits by query rows behind a cached memory budget; carry the same budget
control into our NSA indexer and extend it to CP-ragged topk paths that use
row-wise topk_indices_offset_override.
This keeps the previous one-time cached memory-budget behavior rather than the
recent current-free-mem per-forward variant that regressed performance. A new
optional max-rows env provides an explicit hard cap for debugging or controlled
ETE runs without adding host syncs.
Constraint: DeepGEMM materializes fp32 [q, k] logits internally, so row chunking is the narrowest way to cap temporary memory
Rejected: Restore the reverted syh current-free-mem implementation | it changed hot-path heuristics and showed poor runtime performance
Rejected: Split by K/context dimension | would change topk semantics and require a different transform contract
Confidence: medium
Scope-risk: moderate
Directive: CP-ragged chunking relies on topk_indices_offset_override being row-addressed; do not route non-ragged CP paths through it without separate validation
Tested: Local py_compile for environ.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py
Tested: Remote g0034 cjy-glm5-new py_compile for environ.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py
Tested: Remote pytest TestCpSharedKVTaiMaterializeIntegration, 17 passed
Not-tested: CUDA ETE high-cache-hit bs>1 workload memory/performance after chunking
Co-authored-by: OmX <omx@oh-my-codex.dev>
Temporarily revert the syh MQA-logits row-chunking port because the observed
runtime performance is worse on the current bs>1 CP prefill workload. Keep the
history explicit so the optimization can be revisited after profiling identifies
where the extra overhead comes from.
This reverts commit 4e49751406 (Bound NSA MQA logits peak memory).
Constraint: Current priority is restoring the faster known path for remote ETE runs
Rejected: Keep chunking behind the force/debug env only | the production auto-chunk path still changes runtime heuristics and should not stay until profiled
Confidence: high
Scope-risk: narrow
Directive: Reintroduce logits chunking only with ETE performance evidence and forced-chunk equivalence coverage
Tested: Local py_compile for environ.py and nsa_indexer.py
Not-tested: Remote ETE performance after revert
Co-authored-by: OmX <omx@oh-my-codex.dev>
Paged and CP-ragged NSA indexer paths could materialize q x context fp32
MQA-logits buffers large enough to OOM high-cache-hit bs>1 prefill batches.
Port the syh branch chunking logic so paged and ragged paths split logits by
query rows when the estimated logits buffer exceeds the current free-memory
budget.
The free-memory query is cached on forward_batch so the OOM guard uses current
free memory without adding a torch.cuda.mem_get_info host sync on every layer.
The only new env kept from the syh commits is
SGLANG_NSA_MQA_LOGITS_CHUNK_FORCE_ROWS, which forces chunking for equivalence
validation.
Constraint: DeepGEMM fp8_mqa_logits still materializes fp32 logits internally, so limiting q rows is the least invasive way to cap peak memory
Rejected: Carry unrelated syh envs for page trace/source-fingerprint strictness | not part of the logits peak-memory fix
Rejected: Static mem_fraction-only budget | overestimates logits headroom shared with other forward activations
Confidence: medium
Scope-risk: moderate
Directive: Keep chunking row-split only; changing K/context partitioning needs topk_transform equivalence validation
Related: 40a0389a9c feat(nsa): chunk paged + CP-ragged MQA-logits by current-free-mem budget
Related: 108fa1f538 perf(nsa): cache MQA-logits free-mem budget per-forward
Tested: Local py_compile for environ.py and nsa_indexer.py
Tested: Remote g0034 cjy-glm5-new py_compile for environ.py and nsa_indexer.py
Not-tested: CUDA ETE run with forced SGLANG_NSA_MQA_LOGITS_CHUNK_FORCE_ROWS equivalence check
Not-tested: Full high-cache-hit bs>1 prefill OOM regression workload
Co-authored-by: OmX <omx@oh-my-codex.dev>
New sgl-deep-gemm wheels persist cubins, but SGLang was still replaying the full warmup envelope after every restart. The wrapper also translated SGLANG_DG_* env after importing deep_gemm, so the wheel could observe stale cache settings.\n\nMove DeepGEMM JIT env preparation before the first import, bridge the current SGLANG_DG_USE_NVRTC spelling, replace dense M walks with category-specific sparse M grids, and add a SGLang-side warmup manifest under the DeepGEMM cache dir. A manifest hit skips only the SGLang replay loop; missing or stale cache entries still force warmup and refresh the manifest.\n\nConstraint: sgl-deep-gemm 0.1.2 removed the compile-mode API, so dense 1..m_max loops launch real kernels.\nConstraint: DeepGEMM cache keys include compiler flags such as the deep_gemm include path; path-stable environments are still required for cross-container cache hits.\nRejected: SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 | bypasses warmup instead of making cache reuse correct.\nRejected: Enable per-process cache dirs by default | isolates corruption but defeats persistent cross-process cache reuse.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce dense DeepGEMM warmup for new wheels unless compile-mode support is verified on the target wheel.\nTested: Local py_compile for jit_cache.py configurer.py compile_utils.py model_runner.py\nTested: Local pytest test/registered/unit/layers/test_deep_gemm_jit_cache.py (6 passed)\nTested: Remote g0034 cjy-glm5-new py_compile for same files\nTested: Remote g0034 cjy-glm5-new pytest test/registered/unit/layers/test_deep_gemm_jit_cache.py (6 passed)\nTested: Remote env probe verified SGLANG_DG_CACHE_DIR and SGLANG_DG_USE_NVRTC bridge to DG_JIT_* before configurer completes\nNot-tested: Full decode cold-cache/hot-cache restart timing and manifest-hit ETE log validation
CP shared-KV bs>1 batching was only bounded by request count, extend tokens, and cached tokens. That left temporary GPU buffers such as MLA/index materialization, remap metadata, logits windows, and transfer descriptors implicit, and raw extend-token limits could exceed the active chunked-prefill budget.\n\nThis adds an explicit max-buffer-size admission gate with a CPU-only stream-aware estimator, wires it through PrefillAdder/Scheduler, performs a startup CUDA smoke allocation when configured, and reports the estimate in the scheduler admission benchmark. When chunked prefill is active, the effective CP extend-token limit is capped by the current chunk budget so the CP path does not advertise unreachable batch capacity or lift max-prefill-tokens too far.\n\nConstraint: Admission estimation must stay CPU-only on the scheduler hot path; CUDA allocation is limited to startup smoke checking.\nConstraint: Single oversized requests must still be allowed to run alone to avoid scheduler deadlock.\nRejected: Rely only on --max-prefill-tokens | it does not reliably bound the first oversized request and does not model cache-hit/load-back pressure.\nRejected: Let CP extend limit exceed chunked-prefill size | it creates an unreachable effective capacity and misleading budget lift.\nConfidence: medium\nScope-risk: moderate\nDirective: If bs>1 L1 prefetch is enabled later, update CPSharedKVPrefillBufferEstimatorContext.bs_gt1_l1_prefetch_enabled and include the live prefetch dense buffers in overlap windows.\nTested: local py_compile for touched files\nTested: local PYTHONPATH=python pytest -q test/registered/unit/managers/test_cp_shared_kv_prefill_buffer_estimator.py (4 passed)\nTested: remote cjy-glm5-new targeted pytest for new server_args, PrefillAdder, estimator, and benchmark cases (10 passed)\nTested: remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/managers/test_cp_shared_kv_prefill_buffer_estimator.py test/registered/unit/managers/test_prefill_adder.py test/registered/unit/managers/test_prefill_scheduler_admission_bench.py (29 passed before chunk cap, then test_prefill_adder.py 21 passed after chunk cap)\nNot-tested: full server_args suite because existing TestPrepareServerArgs tries to reach HuggingFace and fails under container DNS/network\nNot-tested: GLM5 ETE smoke with --cp-shared-kv-prefill-max-buffer-size
Add an offline benchmark that reuses PrefillAdder to model how L1 cached tokens, L2 HiCache hits, and extend tokens shape CP shared-KV prefill batch admission. The tool makes scheduler stop reasons and fake L2 load-back capacity pressure observable without starting a model.
Constraint: The benchmark must stay CPU/offline and avoid depending on CUDA execution or live services.
Constraint: L2 cached tokens are modeled as host_hit_length, so successful load-back both increases prefix_len and consumes fake L1 capacity.
Rejected: Build an ETE benchmark first | too slow for isolating scheduler admission behavior.
Rejected: Reimplement scheduler logic from scratch | would drift from PrefillAdder semantics.
Confidence: high
Scope-risk: narrow
Directive: Treat duration_us as Python admission overhead only; it is not an ETE latency metric.
Tested: Remote pytest test/registered/unit/managers/test_prefill_scheduler_admission_bench.py: 4 passed as part of 6 targeted tests.
Tested: Remote synthetic benchmark run with --cp-max-total-cached-tokens showed second 4096-token cached request stopped with OTHER.
Not-tested: Real traffic trace import from production logs.
High cache-hit CP shared-KV batches can have small extend tokens while still carrying substantial prefix/load-back work. Add a CP-specific cached-token admission limit so operators can bound that pressure independently from extend-token batching.
Constraint: The limit must not deadlock a single high-cache-hit request; it only stops adding additional requests to a non-empty batch.
Constraint: Cached tokens are counted after L2 load-back planning via prefix_len, so L1 hits and successful L2 hits share one scheduler budget.
Rejected: Reuse max_prefill_tokens | it limits generic input budget and does not represent cached-token work.
Rejected: Count only L1 prefix before load-back | would miss L2 hit pressure, which is one of the target cases.
Confidence: high
Scope-risk: moderate
Directive: Keep cached-token and extend-token limits separate; they bound different scheduler costs.
Tested: Remote pytest targeted cached-token PrefillAdder cases: 2 passed.
Tested: Remote pytest test/registered/unit/managers/test_prefill_adder.py: 18 passed.
Tested: Remote ServerArgs CP validation smoke: SERVER_ARGS_CACHED_LIMIT_OK.
Not-tested: Full ETE replay with a production cached-token limit value.
CP shared-KV bs>1 uses cp_shared_kv_prefill_max_total_extend_tokens as its grouping admission limit, but the generic max_prefill_tokens budget could still stop batching earlier. Raise only the legacy input-token admission budget for this CP path while keeping allocator-owned capacity checks unchanged.
Constraint: CP shared-KV bs>1 needs large cache-hit batches without relying on the generic max_prefill_tokens default.
Constraint: Allocator capacity must remain enforced by rem_total_tokens, cur_rem_tokens, and prepare_for_extend().
Rejected: Increase server-wide max_prefill_tokens | would change generic scheduler behavior and non-CP paths.
Confidence: high
Scope-risk: narrow
Directive: Do not use max_prefill_tokens as the CP shared-KV bs>1 grouping limit; use the CP-specific total extend token knob.
Tested: Local py_compile for schedule_policy.py and test_prefill_adder.py.
Tested: Remote pytest targeted CP PrefillAdder cases: 5 passed.
Tested: Remote pytest test/registered/unit/managers/test_prefill_adder.py: 16 passed.
Not-tested: Full ETE replay after this scheduler-only change.
init_load_back logged 'loading back N tokens for node M' at INFO for
every request triggering an L2->L1 HiCache load-back (hi_mamba already
had it at debug). Demote and switch both to lazy %-formatting so the
f-string is not built when debug is off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-e2e audit fixes for NSA index-topk sharing (index_topk_pattern):
- Validate the pattern at init: F/S charset only, must start with F
(layer 0 has nothing to share from), and length must equal
num_hidden_layers — a short pattern previously crashed deep in pool
init with an obscure per-layer ValueError, a long one silently
ignored tail characters. Warn when a pattern shadows a configured
index_topk_freq (pattern takes precedence silently otherwise).
- Fail fast when a shared (S) layer receives prev_topk_indices=None:
both gated indexer call sites previously fell through to omitting
topk_indices entirely, running sparse attention without indices and
silently corrupting output if threading were ever dropped.
- Reject pipeline parallelism with index-topk sharing (same rationale
as the existing TBO guard): topk indices are not threaded across
PP-stage boundaries, and each stage's loop resets them to None.
- Gate the CP shared-KV index prefetcher on the next layer having an
index-cache slot: prefetching an S layer's nonexistent buffer
tripped the inactive-layer fail-fast and disabled the prefetcher
for the entire run on the first F->S transition.
- Log the resolved active index layer plan at pool init so an e2e run
can confirm IndexCache is actually on.
Tests: pattern validation cases incl. the GLM-5 reference 78-char
pattern, next_skip == skip-of-next invariant; existing 'C' pattern
test updated to upstream 'F' charset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
filter_kv_indices_for_cp_rank built rank_page_indices =
kv_indices[range_mask] and then ran np.isin(kv_indices,
rank_page_indices) — provably identical to range_mask itself (a value
is in the filtered subset iff it passes the same range test), at the
cost of an extra sort per chunk send under
SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1. Apply the range mask
directly: 30.5 -> 8.1 us per 1024-page chunk. Differential test pins
equivalence against a verbatim copy of the old logic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
stream_output_generation computed a full load snapshot on every
output flush: get_load() walks the waiting/bootstrap/prealloc queues
summing seqlens and builds a DpRequestInfoqOutput per queued request.
The only consumer is the DP load-balancer path in tokenizer_manager,
which forwards it iff dp_size > 1. Gate the computation on the same
condition; at dp_size == 1 the snapshot was computed and shipped for
nothing on every flush.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The [HiCache-write]/[HiCache-load]/[HiCache-collective] and
[CacheCtrl-write] per-operation logs fired at INFO on the scheduler
thread during serving (every write-through, load-back, ack release,
and rate-limited collective summary), eating into stream gaps.
Demote all 29 to debug; one-time [HiCache-draft] init logs stay INFO
and genuine failures stay WARNING (write_backup CP FAILED after
deterministic retry).
Also repair TestHiCacheEvictLoggingLevels, which asserted markers
that no longer exist in this tree (stale list, failing on the
pristine branch): pin the current hot-path markers as debug-only and
extend the check to cache_controller.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
process_batch_result_disagg_prefill did one synchronous
hidden_states[i].cpu().clone() per finished request, and stored
topk_p/topk_index as GPU row views whose D2H then happened one by one
inside MetadataBuffers.set_buf — 3*bs device syncs per finished batch
on the scheduler thread. Gather the finishing rows once and do one
D2H per tensor (3 total); requests hold CPU row views, which set_buf
copies CPU-to-CPU into the metadata buffers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
from_schedule_batch built temperature/top_p/top_k/min_p (+seed) with
4-5 separate list comprehensions and one synchronous H2D copy each,
plus 4 more passes for the is_all_greedy/need_* flags. Collect
everything in a single pass over reqs and upload the float params as
one pinned non-blocking H2D copy (disjoint device views of one
buffer; filter/merge only index and cat, producing fresh tensors, so
the shared buffer is safe), int32 top_k and optional int64 seeds as
their own pinned copies.
B300 (torch 2.11 cu130), scheduler-thread blocking time per call:
bs=8: 38.8 -> 18.2 us (2.1x); bs=32: 1.9x; bs=200: 1.2x
CPU-only construction at bs=200: 2890 -> 1387 us (2.1x).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The EAGLE bigram fallback built 64K-token keys with an index-based
Python comprehension (6.3 ms per call at 64K tokens). zip + islice
runs the pairing loop in C and avoids copying the shifted operand:
3.6-4.2 ms per call (~1.6x). Output is byte-identical (same tuples);
the tai-kernel fast path is unaffected.
Packing bigrams into int64s via numpy was measured and rejected: the
list<->array boxing makes it slower (4.1 ms) than zip until token ids
are numpy end-to-end, and it would change HiCache storage hash inputs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RadixKey.__getitem__ copied the token list on every slice, and the
match/insert tree walks re-slice the remaining key at every node hop,
making a single match O(len * hops) — quadratic for long cached
prefixes. Slices now return O(1) offset-based views over a shared
backing list; the key-match and child-key functions index the backing
list directly so views are never materialized on the hot path.
Keys stored in tree nodes are compacted at every store site (same cost
as the old copying slices), so lock-ref walks, eviction, splits, and
controller-thread reads never observe a key pinning a transient
backing list. Node-key compactness is enforced by a tree-walk test.
Microbenchmark (64K-token full hit, page_size=64):
16-node path: 1.86 -> 0.87 ms/match (2.1x)
128-node path: 9.80 -> 1.19 ms/match (8.2x), insert re-walk 6.9x
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Record the staged design for batch-aware L1 shared-KV prefetch under CP + HiCache. The plan separates L2->L1 load two-layer lookahead from L1 prefetch one-layer lookahead, keeps target first and draft/EAGLE out of the first phase, and calls out that current suffix reuse remains synchronous until the request-aware span contract is extended.
Constraint: Existing CP shared-KV correctness failures came from batch-global remap/req-id ambiguity, so bs>1 prefetch must carry explicit per-request spans and ids.
Rejected: Reuse the bs=1 scalar prefix_pages prefetcher for bs>1 | it aliases request rows and cannot represent mixed prefix/current spans.
Rejected: Make draft/EAGLE prefetch part of the first phase | target path must be validated first before coupling draft lifecycle.
Confidence: medium
Scope-risk: narrow
Directive: Do not open bs>1 L1 prefetch until req-id fail-fast and per-request slot span tests cover both MLA and index paths.
Tested: git diff --cached --check
Not-tested: Runtime implementation; this is a design document only.
High cache-hit NSA sparse attention only gathers rows selected by topk_indices, but the FP8 RAGGED path dequantized the whole materialized K buffer first. Wire the syh in-place topk-only dequant path behind an explicit env gate so the sparse path can dequant referenced rows at their original row ids while keeping topk_indices unchanged. The old full-dequant path remains the default and the verification gate stays available for small correctness checks only.
Constraint: Production runs should enable only SGLANG_NSA_DEQUANT_ONLY_TOPK=1; VERIFY also runs the full-dequant reference and is too expensive for perf tests.
Rejected: Compact/remap topk buffer | data-dependent shape and remap add synchronization/aliasing risk; syh final in-place contract avoids both.
Confidence: medium
Scope-risk: moderate
Directive: Do not enable SGLANG_NSA_DEQUANT_ONLY_TOPK_VERIFY in throughput runs; use it only for small correctness probes.
Tested: python -m py_compile python/sglang/srt/environ.py python/sglang/srt/layers/attention/nsa_backend.py test/registered/unit/layers/test_nsa_dequant_only_topk.py
Tested: git diff --check
Tested: remote g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_dequant_only_topk.py -> 2 passed
Tested: remote g0034 CUDA smoke for tai_kernel.nsa_prefill.nsa_dequant_topk_inplace topk rows matched torch reference
Not-tested: Full GSM8K or replay ETE with SGLANG_NSA_DEQUANT_ONLY_TOPK=1
The CP shared target+draft HiCache budget was still estimating NSA target
bytes per token as if every target layer had an index cache. After index
skip compacts L1 and L2 index buffers to active layers, that stale estimate
kept host/L2 token capacity artificially low even though the physical host
index allocation was already smaller.
Use the pool's index_active_layer_ids when estimating NSA HiCache bytes per
token, with the existing full-layer fallback for pools that do not expose
compact metadata. This keeps the shared target+draft capacity computation
consistent with the actual L2 host layout.
Constraint: Index skip compacts NSA index cache by active logical layers, but MLA KV remains full-layer.
Rejected: Keep full-layer HiCache estimate | preserves correctness but wastes the L2 capacity recovered by compact index allocation.
Confidence: high
Scope-risk: narrow
Directive: Keep HiCache byte estimates aligned with NSATokenToKVPool allocation metadata; otherwise target+draft token budgets will drift from real host memory use.
Tested: Remote container targeted pytest for active-index HiCache estimate and shared-budget tests -> 3 passed.
Tested: Remote container P3-P6 regression suite plus new capacity tests -> 203 passed, 2 subtests passed.
Not-tested: Full ETE replay/GSM8K after this budget correction.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Index skip reduces the number of target layers that own NSA index state,
but PD transfer and HiCache still assumed dense full-layer state buffers.
This change carries explicit state layer IDs through prefill/decode
registration, compacts device and host index buffers to active layers,
and maps logical layer IDs to compact slots on transfer paths.
The PD side fails fast when prefill/decode disagree on NSA state layer
identity instead of silently truncating or copying mismatched buffers.
Host direct tests now use the same CPU-index descriptor contract required
by the TAI cudaMemcpyBatchAsync path, and host registered memory is
unregistered on tensor finalization to avoid stale cudaHostRegister state
across CUDA tests.
Constraint: CP shared-KV with index_topk skip must keep target/draft state identity explicit before compacting buffers
Constraint: Direct HiCache TAI transfer rejects CUDA indices to avoid hidden D2H copies on the control path
Rejected: Keep full-layer L1/L2 index buffers | wastes the memory/bandwidth that index skip is meant to save
Rejected: Infer state buffer order by count only | can silently corrupt cache when active layer sets differ
Confidence: high
Scope-risk: moderate
Directive: Do not compact or reorder NSA state buffers without carrying logical layer IDs through PD registration and validating both sides
Tested: Remote container py_compile for touched runtime files
Tested: Remote container pytest: test_nsa_pool_host_unit.py, test_model_runner_kv_cache_mixin.py, test_cp_shared_kv_transfer_mapping.py, test_pd_state_layer_ids.py, test_cp_per_layer_transfer.py, test_cp_shared_kv_runtime.py -> 200 passed, 2 subtests passed
Not-tested: Full ETE GSM8K/replay after compacted P3-P6 changes
Co-authored-by: OmX <omx@oh-my-codex.dev>
Centralize the IndexCache skip formula and thread the resulting active logical index layers into NSA KV pools. HiCache now skips only the indexer H2D/D2H payload for inactive target layers while preserving per-layer MLA KV transfer, keeping allocation shape unchanged for this phase.
Constraint: P0-P2 must not compact device or host allocation yet; prefill/decode state transfer still has no logical layer-id metadata.
Rejected: Recompute the skip formula separately in mem_cache | formula drift would corrupt cache or waste transfers when offset/pattern settings change.
Rejected: Skip whole-layer HiCache load/backup | MLA KV remains required for every attention layer.
Confidence: medium
Scope-risk: moderate
Directive: Before enabling compact state buffers or compact allocation, add layer-id metadata validation to PD transfer.
Tested: Local py_compile for touched files; remote pytest in g0034 container: test_nsa_index_layers.py and TestNSAIndexerPageIndices, 20 passed.
Not-tested: ETE replay/GSM8K with --nsa-index-topk-freq 4; PD state-transfer compaction remains unimplemented.
Record the phased design for turning NSA index-topk sharing from a compute-only optimization into HiCache and PD-transfer savings. The plan deliberately separates metadata validation from allocation compaction so prefill/decode state-slot mismatches fail before any bandwidth reduction can corrupt cache contents.
Constraint: CP shared-KV cache corruption previously appeared only on cache-hit paths, so state-layer mapping must be explicit before compact transfer or allocation.
Rejected: Compact L1/L2 index buffers first | without state_layer_ids this can silently map different logical layers to the same transfer slot across prefill and decode.
Confidence: high
Scope-risk: narrow
Directive: Do not skip the state_layer_ids phase before compacting NSA index state buffers.
Tested: Documentation placeholder scan and referenced source-path existence check.
Not-tested: Runtime behavior; this commit is documentation only.
Index-topk sharing previously required editing model config or passing raw JSON overrides. Add first-class server args that merge into json_model_override_args before ModelConfig is read, preserving existing JSON overrides while letting launch scripts toggle index_topk_freq directly. Treat offset 0 as unset so wrappers can use 0 for default behavior without injecting a model-config override.
Constraint: Prefill and decode launch commands need the same effective model config without mutating /ssd model files.
Rejected: Require editing config.json | operationally fragile across g0034/g0035/g0036 model copies.
Rejected: Only use --json-model-override-args | works but is too error-prone for frequent launch-command tuning.
Confidence: high
Scope-risk: narrow
Directive: Keep these flags as model-config override shortcuts; apply them before any get_model_config() call.
Tested: Remote pytest in g0034 container for NSA index override parser tests: 2 passed.
Tested: py_compile for server_args.py and test_server_args.py.
Not-tested: Full prefill/decode ETE launch with --nsa-index-topk-freq enabled.
CP shared-KV HiCache transfers are per-layer, so the host layout should match the access pattern instead of forcing page-major strides through every layer. This adds a direct-only layer_page_first layout, routes per-layer KV and NSA index backup/load through the TAI LF<->LPF direct kernels, and keeps storage/page-buffer metadata paths fail-fast until their page-level contract is redesigned.\n\nThe direct controller keeps host indices in caller order for both page_first_direct and layer_page_first because the TAI direct path requires CPU index descriptors and owns descriptor coalescing. All-layer backup intentionally loops over per-layer direct kernels rather than using the sgl-kernel all-layer direct ABI.\n\nConstraint: layer_page_first is currently host-only CP HiCache; storage backends assume page-major contiguous page metadata.\nConstraint: TAI LPF direct kernels require CPU int64 page indices and complete page spans.\nRejected: silently fallback to SM copy when TAI LPF kernels are missing | that hides production performance regressions.\nRejected: support storage page metadata in this commit | LPF requires a layer-page-level storage contract, not a one-pointer-per-page contract.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not enable storage or kernel backend for layer_page_first without redesigning page-buffer metadata and adding remote ETE coverage.\nTested: local py_compile for touched runtime files.\nTested: remote py_compile in g0034 container for touched runtime files.\nTested: remote targeted pytest: 5 passed for parser/storage/layout/move_indices smoke coverage.\nNot-tested: full CP HiCache ETE with --hicache-mem-layout layer_page_first after this commit step.\nNot-tested: combined CUDA roundtrip tests in one pytest process; previous independent runs passed but combined run exposed a host-memory registration lifecycle issue.
SPEC_V2 builds an EAGLE draft input during target prefill. Under CP shared-KV, the target model may expose draft_hidden_states as a CP-local side channel before CP output collection. The v2 path was still passing global hidden_states and never marked the draft input as CP-local, unlike the non-v2 path.
Thread cp_local_hidden_states through the v2 _draft_extend_for_prefill helper and prefer draft_hidden_states when present. This preserves the semantic marker consumed by the static MLP-sync padding guard without changing the non-CP path.
Constraint: Absorb syh 5562937cf only; SPEC_V2 remains opt-in and broader SPEC_V2-on-CP validation is still separate.
Rejected: Infer CP-local hidden from tensor length | tensor length is ambiguous under static padding and bs>1 compute padding.
Confidence: high
Scope-risk: narrow
Directive: Keep EagleDraftInput.cp_local_hidden_states as an explicit semantic contract; do not replace it with shape-based inference.
Tested: Remote g0034 container red test failed before implementation with unexpected cp_local_hidden_states kwarg.
Tested: Remote g0034 container py_compile for eagle_worker_v2.py and test_nsa_cp_utils.py.
Tested: Remote g0034 container pytest target EAGLE marker tests: 2 passed.
Tested: Remote g0034 container pytest test_nsa_cp_utils.py -k eagle: 2 passed, 99 deselected.
Not-tested: Full ETE with SGLANG_ENABLE_SPEC_V2=1 on CP prefill.
CP shared-KV marks all forwards with uses_cp_shared_kv, but TARGET_VERIFY/decode style forwards may legitimately carry no extend prefix page-plan. The CP split helper previously ran the hard page-plan validator before checking context-parallel extend eligibility, so non-extend spec paths could fail before the later no-split decision.
Hoist the eligibility check and only validate page-plan metadata for real context-parallel EXTEND or shared-KV draft extend forwards. Add a regression that TARGET_VERIFY with no prefix metadata returns no-split instead of failing.
Constraint: Absorb syh 7fea88278 only; MQA logits chunk and per-forward budget changes are intentionally excluded.
Rejected: Relax the validator globally | real prefill page-plan violations must remain fail-fast.
Confidence: high
Scope-risk: narrow
Directive: Do not run CP shared-KV page-plan validation for non-context-parallel forward modes without proving those modes own extend_prefix_lens_cpu.
Tested: Remote g0034 container py_compile for utils/test file.
Tested: Remote g0034 container pytest test_nsa_cp_utils.py -k can_cp_split: 8 passed, 92 deselected.
Not-tested: Full ETE speculative non-deepep MoE path.
CP HiCache load-back eviction planning previously recomputed per-owner page counts from node token tensors while scanning evictable leaves. Under shared-KV pressure this can put scheduler-side planning onto an expensive tensor-padding path and stall before load_back can complete.
This stores per-CP-size owner page counts on CP HiCache metadata and uses that CPU metadata for backed/resident CP nodes. Streaming abort handling also accepts int-like status codes so abort responses do not crash on .name/.value access. Temporary debug runbooks remain ignored.
Unnecessary prefill hot-path timing logs were removed before commit; owner-lane eviction now keeps warning-level output for slow planning, insufficient eviction, or remaining deficits only.
Constraint: CP shared-KV cache residency is page-owner based and already records page owners in CpHiCacheNodeMetadata.
Rejected: Keep verbose prefill/owner-lane timing logs | they proved the issue but add hot-path noise after validation.
Confidence: medium
Scope-risk: moderate
Directive: Do not reintroduce tensor-derived owner counting on CP HiCache backed nodes without measuring scheduler CPU/GPU sync cost.
Tested: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py python/sglang/srt/entrypoints/openai/serving_base.py python/sglang/srt/entrypoints/openai/serving_chat.py python/sglang/srt/entrypoints/openai/serving_completions.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py
Tested: git diff --check
Not-tested: Local pytest collection is blocked by missing starlette dependency.
Not-tested: Full ETE after log cleanup; previous pre-cleanup ETE replay reached 136098.82 prompt tok/s without killing prefill.
CP shared-KV compute padding creates per-request lane slots, so valid rows are not a simple prefix/suffix mask. DeepEP MoE was still seeing dummy rows and using scalar non-padded semantics, which let padding participate in gate/topk and corrupted cache-hit tiny-extend inference.\n\nThe fix compacts CP-local valid rows before MoE dispatch and restores the compact output back to the compute-padded row layout before downstream layer communication. The local GSM8K investigation ledger is now removed from the tracked tree and ignored so future debug notes stay local.\n\nConstraint: CP shared-KV compute-padding layout must keep downstream communicator shapes stable.\nRejected: Disable bs>1/current reuse/cache-hit fast paths | hides the semantic bug and loses the intended performance path.\nRejected: Use num_token_non_padded for MoE under compute padding | valid rows are interleaved with dummy lane slots, not suffix-padded.\nConfidence: high\nScope-risk: moderate\nDirective: Do not feed compute-padded dummy rows into sparse MoE gate/topk; compact valid rows at the MoE boundary and restore shape afterward.\nTested: python -m py_compile python/sglang/srt/layers/attention/nsa/utils.py python/sglang/srt/models/deepseek_v2.py\nTested: remote focused CP utils tests passed, 4 tests.\nTested: remote GSM8K 50-question smoke accuracy 0.960; 200-question runs accuracy 0.955 and 0.965; full 1319-question run accuracy 0.952.\nNot-tested: Long-running production traffic beyond GSM8K after this commit.
The SYH DeepGEMM pick raised the global sglang-kernel runtime guard to 0.4.3, but this deployment and pyproject intentionally use sglang-kernel 0.4.0 while DeepGEMM compatibility is handled by the deep_gemm_wrapper import/call path and tai-provided runtime pieces. A global 0.4.3 guard blocks startup before those targeted checks can run.\n\nConstraint: Do not update pyproject.toml because local and remote environments intentionally differ.\nRejected: pip install sglang-kernel --force-reinstall | mutates the remote environment and conflicts with the pinned project environment.\nRejected: keep a global 0.4.3 guard | blocks the current CUDA deployment despite pyproject requiring 0.4.0.\nConfidence: medium\nScope-risk: narrow\nDirective: Do not raise this global guard without also updating the environment contract and pyproject together; feature-specific wheel/API checks belong near their import/call sites.\nTested: local py_compile for python/sglang/srt/entrypoints/engine.py; remote py_compile in g0034 container; remote installed sglang-kernel version confirmed as 0.4.0.\nNot-tested: full launch_server restart after the guard change.
The SYH per-request page_inverse migration fixes bs>1 cache contamination, but the conflict resolution made several helper/reference paths require explicit request ids and broke existing unit coverage. This keeps production bs>1 callers on explicit req-id routing while allowing bs=1/reference helpers to infer or default request ids without reintroducing the hot-path batch-global inverse.
Constraint: Runtime bs>1 materialize paths must route through per-request page_inverse rows to avoid cross-request KV aliasing.
Constraint: The remote test container may still have single-row tai-kernel materialize helpers while production bs>1 requires the new req-id ABI.
Rejected: Revert to batch-global page_inverse | it is the GSM8K cache-hit corruption root cause.
Rejected: Update tests only | the helper API is still useful for bs=1/reference callers and py-level regression coverage.
Confidence: medium
Scope-risk: moderate
Directive: Do not remove explicit loc_req_id/current_req_id from production bs>1 call sites; default inference is for legacy/reference use only.
Tested: Local py_compile for cp_shared_kv_runtime.py and nsa_indexer.py.
Tested: Remote container py_compile for cp_shared_kv_runtime.py.
Tested: Remote container pytest: test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py test/registered/unit/layers/test_nsa_cp_utils.py test/registered/unit/mem_cache/test_cp_shared_kv_layout.py => 263 passed, 5 warnings, 2 subtests passed.
Not-tested: Full ETE GSM8K/cache-hit run after SYH pick.
Two correctness fixes in the bs>1 CP shared-KV prefill path, validated end-to-end on B300 (GLM-5.1-FP8, EAGLE, fp8 KV, CP8):
1) Cross-request KV contamination: the value-keyed 1-D page_inverse was last-writer-wins, so two requests sharing a physical page (radix shared-prefix / current-reuse) aliased each other's KV. Replace with a per-request flat page_inverse[batch_rows*capacity] indexed req_id*capacity+lp, threaded through build/remap/fill_current/prefetch and the TAI/Triton kernels. req_id sourced via slot//pages_per_request (build), repeat_interleave (logical_locs), and a companion tensor through the same CP valid-split (current_locs).
2) CP+EAGLE compute-padding cache-write: forward_absorb_prepare's rebuild_cp_kv_cache all-gathers current k back to global rows under current-reuse, but the compute-padding branch fed that full k straight to select_cp_local_valid_rows_for_cache_write (which requires per-rank compute rows) -> fail-fast. Add cp_localize_current_kv_to_compute_rows to re-localize via the batch-plan compute split first (mirrors the non-padding branch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 28efef91b05ede1a7072ee451e2aea39ecc3a5bd)
Shrink the non-grouped (dense/attention) DeepGEMM warmup M grid by attn_cp_size under NSA prefill in-seq CP (per-rank M = tokens/cp), while keeping the grouped MoE GEMM grid full (deepep all-to-all re-gathers all tokens; topk==ep_size keeps MoE M ~= chunked). Gated by _cp_dense_warmup_divisor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 672ef3a6609abd100e5e95b42016d17d0a2966e5)
calculate_mla_kv_cache_dim only packs fp8 (override dim -> nsa_kv_cache_store_fp8=True, required by the flashmla_sparse dequant path) when NEITHER prefill nor decode backend is trtllm. The Blackwell-no-DP guard defaulted the unset side to trtllm, producing a mixed flashmla/trtllm pair that left fp8 KV unpacked and crashed flashmla_sparse ('kv must have dtype bf16'). Now an explicit flashmla choice on either side pulls the unset side to flashmla too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit ed30312c3d599595e40ae99bc2832159b634cca6)
