Infer the FP4 weight scale group size from the loaded scale tensor instead of hard-coding K/32.
This keeps upstream-style FP4 expert layouts working while allowing GLM/ModelOpt NVFP4 layouts that use K/16 scale columns to build MegaMoE sidecar weights.
Constraint: preserve the existing runner layout and only change MegaMoE sidecar metadata/recipe.
Feature-flag: --moe-a2a-backend=megamoe.
Conflict-hotspots: python/sglang/srt/layers/moe/mega_moe.py.
Scope-risk: actual DeepGEMM recipe support still needs target GPU runtime validation.
Tested: PYTHONPYCACHEPREFIX=/private/tmp/sglang_pycache python3 -m py_compile python/sglang/srt/layers/moe/mega_moe.py test/registered/unit/moe/test_glm_megamoe.py.
Tested: git diff --check.
Not-tested: GLM 5.2 MegaMoE GPU e2e; local environment lacks target runtime and hardware.
Add the GLM MoE MegaMoE fast path before the existing DeepEP/normal branch selection.
Keep the existing none+flashinfer_trtllm path as the miss path when MegaMoE weights, envs, or token caps do not satisfy should_use_mega_moe().
Match upstream MegaMoE shared-expert TP1 semantics for the separate shared expert path.
Constraint: do not change the baseline none+flashinfer_trtllm dispatch path.
Feature-flag: --moe-a2a-backend=megamoe.
Conflict-hotspots: python/sglang/srt/models/glm4_moe.py.
Scope-risk: shared-expert TP1 behavior matters when shared expert fusion is disabled.
Tested: PYTHONPYCACHEPREFIX=/private/tmp/sglang_pycache python3 -m py_compile python/sglang/srt/models/glm4_moe.py.
Tested: git diff --check.
Not-tested: GLM 5.2 MegaMoE GPU e2e; local environment lacks target runtime and hardware.
Add a shared MegaMoE forward path and expert weight builder that can be used by GLM MoE without DeepSeek-only type assumptions.
Build ModelOpt NVFP4 MegaMoE sidecar weights before FlashInfer TRTLLM alignment so the original runner layout remains available as fallback.
Keep the runtime fast path gated to DeepGEMM's FP4 activation pre-dispatch because this fork does not carry the upstream DSV4 JIT FP8 pre-dispatch kernels yet.
Constraint: preserve none+flashinfer_trtllm runner layout as fallback.
Feature-flag: --moe-a2a-backend=megamoe and SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS.
Conflict-hotspots: python/sglang/srt/layers/moe/mega_moe.py, python/sglang/srt/layers/quantization/modelopt_quant.py.
Scope-risk: DeepGEMM runtime APIs and GPU e2e are not available on this local machine.
Tested: PYTHONPYCACHEPREFIX=/private/tmp/sglang_pycache python3 -m py_compile python/sglang/srt/layers/moe/mega_moe.py python/sglang/srt/layers/quantization/modelopt_quant.py.
Tested: git diff --check.
Not-tested: GLM 5.2 MegaMoE GPU runtime; local environment lacks deep_gemm and target GPU.
Add the MegaMoE A2A backend name to server argument choices and MoE backend enums, wire the startup handling that forces ep_size to tp_size, and add the env flags used by the later MegaMoE forward and weight layout work. Treat MegaMoE as a dispatcher bypass so it does not enter DeepEP dispatcher logic.\n\nConstraint: Applies on top of origin/rc/v0.1 for GLM 5.2 MegaMoE adaptation.\nFeature-flag: --moe-a2a-backend megamoe and SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE.\nConflict-hotspots: python/sglang/srt/server_args.py; python/sglang/srt/layers/moe/utils.py; python/sglang/srt/layers/moe/fused_moe_triton/layer.py\nScope-risk: medium\nTested: git diff --check\nTested: python3 -c ast.parse on modified Python files
The old min_new_tokens penalizer updated logits through boolean-mask indexing. That indexing is data-dependent and can force synchronization on the decode hot path.
Use an elementwise torch.where followed by inplace add so the operation stays tensorized and avoids the mask-index update path.
Constraint: Keep the numeric behavior for active rows equivalent without multiplying by zero, which would turn -inf penalties into NaN.
Rejected: Expanding the mask and assigning through logits[mask] | this is the synchronization pattern being removed.
Confidence: high
Scope-risk: narrow
Directive: Do not reintroduce boolean-mask writes in decode-step penalty paths without profiling the synchronization behavior.
Tested: RED/GREEN local pytest test/registered/unit/sampling/test_min_new_tokens_penalizer.py
Tested: RED/GREEN remote pytest in cjy-glm5-new for min_new_tokens penalizer together with related regression tests
Tested: git diff --check; py_compile for min_new_tokens.py
Not-tested: Full decode throughput benchmark
ReasonerGrammarObject wraps an inner grammar, but disaggregated prebuilt replay checks the wrapper current_token to avoid accepting an already accepted token twice. Track the token on the wrapper itself so reasoning grammar follows the same contract as XGrammar.
Speculative decode can accept a stop string and EOS in one step. Check stop strings before token-based EOS after sanitizing invalid token ids, and set finished_len at the matched stop position so trailing accepted tokens do not leak.
Constraint: Current branch predates upstream helper methods for locating string stop positions, so the stop-string fix is manually ported instead of cherry-picked.
Rejected: Direct cherry-pick of bbc853df46 | current schedule_batch.py lacks the upstream helper context.
Confidence: high
Scope-risk: moderate
Directive: Keep vocab-boundary sanitization before string decoding; do not move token-based EOS ahead of stop-string checks without a same-step speculative regression test.
Tested: RED/GREEN remote pytest in cjy-glm5-new for constrained current_token and stop-string speculative tests
Tested: git diff --check; py_compile for reasoner_grammar_backend.py and schedule_batch.py
Not-tested: Full scheduler/disaggregation integration suite
Decode prealloc and transfer queues own the receiver lifetime once a request leaves the queue. The abort and transfer-failure paths were removing requests after streaming/releasing KV state without clearing the receiver, leaving backend-specific request tracking behind. The scheduler idle check also ignored decode retracted requests, so idle housekeeping could run while decode handoff state still existed.
Constraint: This is a manual port of upstream 18989f3d48 onto a locally diverged decode queue implementation.
Rejected: Direct cherry-pick | current decode queue compaction and streamer APIs differ from upstream.
Confidence: high
Scope-risk: narrow
Directive: Any path that removes a DecodeRequest from prealloc or transfer ownership must clear its kv_receiver unless ownership is explicitly transferred.
Tested: remote cjy-glm5-new pytest test/registered/unit/disaggregation/test_decode_queue_compaction.py -q: 16 passed
Tested: remote cjy-glm5-new pytest test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/observability/test_scheduler_metrics_load.py -q: 17 passed
Tested: git diff --check on modified files
Not-tested: full disaggregated decode E2E under live traffic
Spec v1 EAGLE now follows the same model-config-gated MTP index reuse contract as spec v2. This lets models that declare index_share_for_mtp_iteration reuse the first draft step's NSA/DSA topk indices across internal MTP iterations while keeping the unsafe topk>1 case disabled.
Constraint: select_top_k_tokens can reorder hidden rows when topk > 1, so carried topk indices are only valid under topk == 1.
Rejected: Enable reuse unconditionally | models without the config flag may not have compatible MTP index semantics.
Rejected: Broaden to target-to-draft index reuse | separate semantic change with different correctness risks.
Confidence: high
Scope-risk: narrow
Directive: Keep spec v1 and spec v2 MTP index reuse semantics aligned, including the topk==1 guard and per-draft-forward cleanup.
Tested: python -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q
Tested: python -m py_compile python/sglang/srt/speculative/eagle_worker.py test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py
Not-tested: full spec v1 GLM5 online throughput run
EAGLE V2 draft MTP can avoid recomputing NSA/DSA indexer topk across internal draft iterations when the model declares that those indices are shareable. The port follows the upstream narrow contract: store per-forward topk on ForwardBatch, enable reuse only for topk=1, and clear the transient state after draft_forward.
Constraint: topk > 1 reorders hidden rows in select_top_k_tokens, so carried topk indices would no longer match the hidden states.
Rejected: Reuse target-side topk for draft | broader semantic change not covered by the upstream fix or current tests.
Rejected: Skip loading draft indexer weights | separate memory optimization with correctness risk for models that do not enable MTP index sharing.
Confidence: high
Scope-risk: narrow
Directive: Do not enable index_share_for_mtp_iteration without preserving the topk==1 guard and per-draft-forward cleanup.
Tested: python -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q
Tested: python -m py_compile python/sglang/srt/speculative/eagle_worker_v2.py python/sglang/srt/models/deepseek_nextn.py python/sglang/srt/model_executor/forward_batch_info.py test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py
Not-tested: full GLM5 EAGLE throughput/accuracy run
NSA spec-v2 draft-extend graph replay was still using host-derived sequence lengths and the draft-decode backend allowlist. That kept NSA on eager draft-extend for spec-v2 and left seq_lens_cpu.max()/list-to-GPU tensor construction on the decode critical path.
This ports the small upstream DSA metadata fixes into the local NSA backend: size the captured graph page table to req_to_token width, use the static captured page-table width for graph replay, split v2 draft-extend from variable-length v1 draft-extend, and decide draft-extend graph support from the prefill-style backend.
Constraint: Current branch does not have the full upstream needs_cpu_seq_lens scheduler/FutureMap infra.
Rejected: Cherry-pick the full DSA fused metadata generation series | too broad and overlaps with local NSA fused metadata-copy code.
Confidence: medium
Scope-risk: moderate
Directive: Do not collapse DRAFT_EXTEND and DRAFT_EXTEND_V2 here; v1 keeps variable accept lengths while v2 must stay graph-static.
Tested: local pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q (19 passed)
Tested: local py_compile on nsa_backend.py, nsa_backend_mtp_precompute.py, eagle_worker_v2.py
Tested: remote g0034 cjy-glm5-new pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q (19 passed)
Tested: remote g0034 cjy-glm5-new py_compile on nsa_backend.py, nsa_backend_mtp_precompute.py, eagle_worker_v2.py
Not-tested: full decode E2E with SGLANG_ENABLE_SPEC_V2=1
Spec-v2 decode was paying unnecessary scheduler-side copy and replay costs: hidden states were copied back to CPU even when not requested, overlap result D2H ran on the forward stream, and draft graph replay issued several small copies separately.
Port the focused upstream optimizations without taking the larger spec-v2 relay refactor: gate hidden-state D2H on return_hidden_states, run result copies on copy_stream with pinned async D2H, use stride arange for draft select_index, and group small draft graph replay copies while keeping the large hidden-state DMA copy separate.
Constraint: Current branch carries local GLM/NSA/PD/CP changes, so upstream spec-v2 large refactors are not safe to cherry-pick wholesale.
Rejected: Cherry-pick upstream spec-v2 relay/dataclass refactor | too broad for this performance fix and conflicts with current branch structure
Rejected: Copy hidden states unconditionally | default chat output does not consume them after draft extend
Confidence: medium
Scope-risk: moderate
Directive: Keep result-copy lifetime tied to copy_done; do not move copy_to_cpu back onto forward_stream without measuring decode throughput.
Tested: local py_compile for changed production files and new test
Tested: local git diff --check
Tested: remote g0034 cjy-glm5-new as ubuntu on /sgl-workspace/sglang-tai: pytest -q -p no:cacheprovider test_generation_batch_result_copy.py test_eagle_worker_v2_cp_hidden.py test_spec_utils.py
Not-tested: full two-node decode throughput A/B after restart
Spec-v2 draft extend can receive token ids from producers whose dtype is not already int64, while DP collective paths require a stable integer dtype across ranks. EAGLE draft CUDA graph replay also pads raw batches to a captured batch size, so the metadata/replay path must see seq_lens_sum consistent with the padded seq_lens and then restore the caller-visible raw value.
Constraint: Keep this as a narrow correctness port from upstream rather than pulling the larger spec-v2 refactor chain.
Rejected: Cherry-pick broader attention-backend and decode-result refactors | current branch lacks the same upstream forward-context scaffolding and would require a separate port.
Confidence: high
Scope-risk: narrow
Directive: Do not remove the seq_lens_sum restore without rechecking padded EAGLE draft CUDA graph metadata construction.
Tested: python -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q
Tested: remote g0034/cjy-glm5-new PYTHONPATH=python python3 -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q
Not-tested: full multi-node GLM5 spec-v2 decode startup smoke
Co-authored-by: OmX <omx@oh-my-codex.dev>
Port the fix-decode spec-v2 ownership and plan-stream fixes onto the current branch. Draft extend now keeps the scheduler-owned batch lengths committed until acceptance, binds the draft runner to the draft-extend attention backend, keeps speculative KV allocation monotonic, and lets non-graph target verify initialize metadata after DP padding. The worker also records rebound tensors on the forward stream and orders plan-stream metadata work after current-stream inputs are available.
Constraint: fix-decode commits cd8e47ed9c and 60e3956d9c address CUDA illegal-address failures in spec-v2 decode warmup paths.
Rejected: Cherry-pick the commits blindly | the current branch has intervening decode changes, so a minimal manual port kept the patch surface to the affected files.
Confidence: medium
Scope-risk: moderate
Directive: Do not move target-verify non-graph metadata initialization back into prepare_for_v2_verify without validating DP padding and NSA metadata ordering.
Tested: RED local pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q failed 8/8 before production changes.
Tested: PYTHONPATH=python python3 -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q passed 8/8 locally.
Tested: PYTHONPATH=python python3 -m py_compile python/sglang/srt/speculative/eagle_info_v2.py python/sglang/srt/speculative/eagle_worker_v2.py python/sglang/srt/speculative/spec_utils.py passed locally.
Tested: Remote g0034:cjy-glm5-new pytest for test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py passed 8/8, plus remote py_compile for the three speculative modules.
Tested: Local and remote sha256 sums matched for all four synced files.
Not-tested: Full spec-v2 decode server restart/warmup under production traffic.
Under CP, attn_cp_size shards the TP group so compute_dp_attention_world_info
collapses attn_tp_size to 1 -> attn_tp_rank == 0 on EVERY CP rank. So
is_stats_logging_rank and current_scheduler_metrics_enabled were True on all
attn_cp_size ranks, making each rank log a 'Prefill batch' line and emit
scheduler-side metrics (SchedulerStats gauges, realtime token throughput) ->
summed token-throughput inflated by cp_size and duplicated per-rank logs.
Add attn_cp_rank == 0 to both gates, matching the scheduler's other single-rank
guards (scheduler.py:633/1591/1615/1631). No-op when CP is off (attn_cp_rank is
always 0). init_metrics runs after attn_cp_rank/attn_tp_rank are set
(scheduler.py:375/415 -> 433). Does NOT affect sglang:num_requests_total (a
TokenizerManager counter, one process per engine).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rank-local CP HiCache async write queues can drain at different times across ranks. Radix structural decisions that consult those local queues can diverge, so backup-pending state now lives on the replicated radix node and uses a separate backup lock ref from request locks. Commit and rollback clear the replicated marker, and split/prune/hit-count probes use the marker rather than local pending dictionaries.
Constraint: CP ranks must make identical radix structural decisions even when local async write acknowledgements drain at different times
Rejected: Drain write acknowledgements before every split/prune decision | still rank-local and adds hot-path synchronization
Rejected: Treat backup lock_ref as a normal request lock | hides the distinction between request protection and backup-pending protection
Confidence: high
Scope-risk: moderate
Directive: Do not derive CP HiCache split/prune/write-through decisions from pending_host_backups or ongoing_write_through without a replicated node marker
Tested: local py_compile python/sglang/srt/mem_cache/hiradix_cache.py python/sglang/srt/mem_cache/radix_cache.py
Tested: local git diff --check
Tested: remote cjy-glm5-new pytest replicated marker test plus targeted CP HiCache metadata subset, 10 passed
Not-tested: full test_cp_hicache_metadata.py because current branch has unrelated fixture drift around enable_cp_l3/cp_shared_l2_page_allocator/running-count setup
Co-authored-by: OmX <omx@oh-my-codex.dev>
Attribute decode CUDA-graph GPU memory at capture time, gated by
SGLANG_LOG_CG_BUFFERS=1 (off by default, startup-only, rank0):
1. Per-runner static input/output buffer table (name/shape/dtype/MB)
across the 3 EAGLE families (target-verify, draft, draft-extend),
flagging buffers that share_buffers() aliased onto an earlier family.
2. Per-shape graph-pool growth deltas in the capture loop + a per-family
TOTAL, isolating the shared activation pool from per-shape outputs.
Also fix --enable-profile-cuda-graph dumping all families to one
overwritten cuda_graph_runner_memory_usage.pickle: now writes a unique
cuda_graph_mem_<family>.pickle per runner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_drain_l3_control_queues runs validate-candidates -> admit -> reserve in one tick.
_cp_l3_admit_reload pops the inflight key and mark_object_committed()s the l3rl
object (which stays live in the allocator) while inserting the reloaded node. A
second same-suffix candidate, validated against the pre-admit tree, then reaches
_cp_l3_reserve_reloads with its dedup guard missing (inflight key gone) and calls
allocator.reserve() on the still-live object -> ValueError: duplicate live
reservation for object 'l3rl:<hash>' payload 'target_kv' -> scheduler crash.
Re-validate _cp_l3_reload_attach_ok in _cp_l3_reserve_reloads after the piggyback
check (honoring its documented 'checked at reserve AND at admission' contract,
which the original code never enforced). If a same-tick admit inserted the suffix,
the candidate is dropped; the un-held request re-matches and hits the just-inserted
L2 node. Rank-uniform: attach_ok reads only the replicated radix and admit is
rank-uniform, so every CP rank drops the same candidate. The reserve() duplicate
raise is kept as a fail-loud backstop for true invariant violations.
Confirmed by independent root-cause + adversarial fix review (CLEAN-SHIP): the
split-retained-key path that looks residual is provably unreachable via the radix
content-path / match_prefix invariant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the O(victims*candidates) per-iteration greedy argmin rescan in
_plan_cp_load_back_owner_lane_evictions with a leaf-up min-heap. The score
(-contribution, -unlock, slru_priority, node.id) depends on the current deficits, so a
static heap is not equivalent; instead use LAZY RE-EVALUATION: a popped entry is consumed
only if its score still matches the node's current score (deficits unchanged since push),
else it is re-pushed with the fresh score. Stale scores are always optimistic (contribution
= sum(min(counts[o], deficits[o])) and the ancestor-unlock contribution only shrink as
deficits shrink), so the first up-to-date popped entry is exactly the global argmin the full
rescan would have picked -> PROVABLY EQUIVALENT, with the same leaf-up eligibility +
ancestor-unlock + parent-push. Determinism/rank-uniformity preserved (selection decided by
the total-ordered score; the heap insertion seq only orders structurally-equal tuples).
Equivalence proven by test: a verbatim reference greedy + a randomized property test (300
seeds, single- AND multi-owner deficits, non-uniform counts exercising the lazy-re-eval
boundary, varied SLRU priorities) asserting byte-identical (victims, planned_freed,
remaining), plus an explicit leaf-up + ancestor-unlock case (child-then-parent). 14 planner
tests pass (1 pre-existing unrelated EAGLE-tail failure unchanged).
Micro-bench (V100, candidates=2000 deficit=7877, benchmark/hicache/bench_cp_owner_lane_planner.py):
A. ORIGINAL (.item() x cp, no memo) 50.06s
B. memo + bincount (greedy) 0.494s (101x)
C. lazy-re-eval heap 0.162s (309x; 3.0x over B) selection A==B==C identical
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A CP4 prefill server hung 79s in _plan_cp_load_back_owner_lane_evictions (the [HiCache-load]
slow-scan) and was killed by the detokenizer health-check. Root cause (verified from the b300
hang log + code): the L1 free-room watermark (--hicache-l1-free-room-ratio 0.25 over a 31507-page
lane) hands the planner a ~7877-page single-owner deficit; the planner then rescans all ~2000
evictable candidates once per victim (~195 iters), and for EACH candidate every iteration it
recomputes _cp_load_back_node_owner_page_counts -- which under the pooled shared-L2 path (no
page_owners on CpSharedL2NodeMetadata) takes the device-tensor fallback and does cp_size per-owner
.item() device->host syncs. That is ~195 * 2000 * 4 ~= 1.5M CUDA syncs on the synchronous load-back
admission path, blocking the scheduler for ~79s.
Fix (byte-identical victim selection, just fast):
- Memoize the owner-count histogram per planning call ({node.id: counts}); the counts are invariant
while node.value is fixed (the plan does not mutate values), so node.id is a safe key for the plan's
duration. Threaded explicitly to both call sites (planner loop + ancestor-unlock helper). Turns
O(victims*candidates) recomputes into O(distinct nodes).
- Replace the cp_size per-owner sum().item() loop with one bincount().tolist() device->host sync.
Net: ~79s -> ~1s; the eviction plan (victims, planned_freed) is unchanged. bincount == the per-owner
loop proven over 8000 random vectors incl. the (-1)%cp==cp-1 zero-loc edge. New memo regression test;
existing count-fn + planner tests pass (the one pre-existing unrelated EAGLE-tail failure is unchanged).
(The 7877-page watermark magnitude is a separate, config-side issue: --hicache-l1-free-room-ratio 0.25
reserves ~25% of a ~2M-token lane -- ~30x more than a 64K chunk needs; lower it.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The B1 placement-replication assert (SGLANG_CP_HICACHE_PLACEMENT_ASSERT) MIN/MAX-reduces a SHA256
placement_digest across the CP group EVERY scheduler tick, on the scheduler thread, BEFORE the forward
launch (2 blocking gloo collectives + an O(free-ranges+objects) hash per tick). It is debug/bring-up
only. The census (docs_internal/cp_hicache_perf_investigation_2026-06-23.md) shows it ~quadruples the
per-tick collective count vs production; the b300 wire-test script sets it =1, so it is the prime
suspect for "poor perf even L2-only". Warn once so an accidental prod run with it on is visible.
No behavior change when off (cached-bool early-return).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On restart the LMDB index + disk-slab blobs persist, but CpL3Store.from_config built the slot
pools all-free and the GC LRU empty -> the next spill re-hands-out a slot the durable index still
references (clobbers a live blob) and GC never reclaims the carried-over entries. Neither a clean
start nor a durable reload was actually realized.
connect() now applies a cold-start policy BEFORE the bg threads start (the write thread is the sole
pool/GC owner, so the single-threaded reconcile must precede it):
- clear (default): wipe the persisted index + reset the pools/GC -> genuinely empty start (disk
blobs are inert, overwritten lazily on slot reuse).
- load: rebuild this rank's slot free-list + GC LRU from the durable disk blobs. Drive the scan
from the rank's OWN slab file (header-only reads) so it never inspects another rank's slots even
when ranks share a disk; a slot is LIVE iff its blob header parses AND the shared index still maps
that content hash back to this exact slot -> occupy + seed the GC LRU with the durable last_access;
orphan/unwritten slots stay free. Header-only (no payload CRC); reload-time verify-on-read still
fail-softs a torn payload. The L3 durable floor now survives a process restart.
Primitives: CpL3SlotPool.rebuild_from_allocated (O(num_slots) bulk-occupy) + CpL3DiskSlab.read_header
(one aligned block, not the multi-MB slot). Env SGLANG_CP_L3_COLD_START (default "clear"), read in
_maybe_init_cp_l3 and passed to connect(). Tests: 4 cold-start e2e (load rebuilds the floor + no slot
collision after a fresh spill; GC LRU rebuilt; clear starts empty; unknown mode fails loud) + 2 unit
(rebuild_from_allocated, read_header). 23 L3 store/disk/posix tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A radix split crashed: split_committed_object got split_pages=172 (from the node
KEY length) for a target_kv object of num_pages=1 (the host backup), raising
"split page count must leave at least one page on each side". Root cause: the L3
reload inserts a MALFORMED node -- _cp_l3_reserve_reloads stores the FULL n_full-
page suffix key but only the C L3-durable pages are reserved/hashed/host-backed
(C < n_full when L3 holds a partial durable prefix via page-level GC or partial
spill). The node then has len(key)//page = n_full but host_len//page =
object.num_pages = len(hash_value) = C, violating the radix invariant every
consumer assumes; the split derives split_pages from the long key and overruns the
C-page object. Pre-existing (L3 3.2, 864b1c808e); NOT the multi-slab change
(_allocate_contiguous raises on no-fit, never partial, so multi-slab cannot produce
a partial-backed node -- it crashes single-slab too).
Elegant fix -- restore the invariant at its single source: slice the reloaded node
to its C durable pages so len(key)//page == host_len//page == object.num_pages ==
len(hash_value) == C. The un-durable suffix [C:n_full] is a correct cache miss.
This makes _split_node / CpSharedL2NodeMetadata.split / split_committed_object / the
L3 page-set builders correct unchanged, and removes the latent negative-host_len and
value/hash mis-slice the same malformed node would have triggered.
- hiradix_cache.py _cp_l3_reserve_reloads: slice suffix_key to C pages (matching the
already-sliced suffix_hashes).
- hiradix_cache.py _cp_l3_build_owned_pages (spill) + _cp_l3_build_reload_owned_pages
(reload): fail loud if the page-hash count exceeds the object's pages (else a future
desync silently corrupts a neighbour object in the same slab -- the slab accessor
only catches a whole-slab overrun).
- hiradix_cache.py _split_node: assert split_len <= host_len (fail loud on any future
malformed partial-backed node instead of the cryptic split_pages>=num_pages).
- test_cp_shared_l2_pool.py: lock the split contract (split_committed_object rejects
split_pages outside (0,num_pages); metadata.split rejects object/padded mismatch).
Design + 3-agent investigation + 2-agent review (no bugs):
docs_internal/cp_hicache_l3_reload_partial_prefix_fix_design.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_cp_maybe_collect_metrics read cache_controller.tp_rank/dp_rank, but those were
resolved only inside _generate_storage_config -- which never runs under CP
shared-L2 (storage backends are forbidden there), so the first metrics emit
crashed the prefill scheduler with:
AttributeError: 'HiCacheController' object has no attribute 'tp_rank'
- cache_controller.py: resolve tp_rank/tp_size/dp_rank once in __init__ (always
available, not storage-gated); drop the now-redundant derivation in
_generate_storage_config.
- hiradix_cache.py: label the CP HiCache metrics by cp_rank/cp_size -- the
identity that actually distinguishes the per-rank L2 lanes / L3 slabs these
metrics measure -- resolved from the CP layout (_cp_hicache_cp_*), alongside
the now-reliable tp/dp/pp coordinates.
- metrics_collector.py: docstring -> per-rank via cp_rank.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A single cudaHostRegister over a >~1 TB host buffer fails with
cudaErrorMemoryAllocation on B300 (hard per-call ceiling: 512 GiB OK, 1024 GiB
FAIL), crashing the hicache_size=1600 (~1.5 TB CP shared-L2 slab) prefill at
startup. The registration cannot simply be chunked: a memcpy (cudaMemcpyBatchAsync,
the CP-L2 H2D/D2H transfer) fails with cudaErrorInvalidValue when its host range
straddles a registration boundary (verified empirically on b300-049).
Fix: physically split the host cache into multiple page-aligned slabs, each <= a
safe single-registration size (default 480 GiB, env SGLANG_CP_HICACHE_MAX_SLAB_GB),
reusing the existing SharedHostTensorGroupAllocator + per-slab transfer splitting
(_host_transfer_segments). Each slab is one whole registration and no transfer
crosses a boundary; small configs (hicache_size<=400) stay single-slab unchanged.
- memory_pool_host.py: add cp_hicache_max_single_register_bytes() + the fail-loud
_check_single_cuda_host_register_size guard; revert the (transfer-unsafe)
registration chunking back to one cudaHostRegister per buffer/slab.
- hiradix_cache.py: _cp_shared_l2_slab_pages_by_payload auto-caps each payload's
slab <= the ceiling so large caches auto-split.
- cp_l3_slab_accessor.py: CpSharedL2SlabAccessor is now slab-count-aware
(CpL3SlabSpan + per-slab dispatch via global_base_page; per-slab layer stride);
_cp_l3_slab_spans rewires _maybe_init_cp_l3 off the single-slab assumption. L3
disk slabs / slot pool / LMDB index / GC are content-addressed and unchanged.
Tests: multi-slab accessor incl. a non-circular torch.frombuffer-layout check; a
real allocate_group + _cp_l3_slab_spans roundtrip; slab-cap auto-split; L3 store
cross-slab spill/reload. Reviewed by 3 adversarial agents, no correctness bugs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The host-pool size check used psutil.virtual_memory().available -- regular RAM -- unconditionally. But
hugepages are carved OUT of regular RAM, so reserving e.g. 1 TB of 2M hugepages drops psutil.available
by ~1 TB, and the check then spuriously fails ("Not enough host memory ... only have 897 GB free") even
though the slab is allocated from the hugepages, not regular RAM. It was inspecting the wrong pool.
Two fixes:
- memory_pool_host.py: gate the psutil check on the DEFAULT (regular-malloc) allocator. A custom
host_tensor_allocator (the CP shared-L2 slab over hugetlbfs) owns its own capacity check, so skip the
regular-RAM check for it.
- cp_shared_l2_pool.py: add cp_shared_l2_hugetlbfs_free_bytes() + check_cp_shared_l2_hugetlbfs_capacity()
(statvfs the hugetlbfs mount: f_frsize = hugepage size, f_bavail = free hugepages) and call it in
create_cp_shared_host_slab before ftruncate/mmap. Now insufficient hugepages fail LOUD with a clean,
actionable message ("insufficient hugepages in /dev/hugepages: need X GB (N x 2M pages) but only Y GB
free; reserve more or reduce --hicache-size") instead of a cryptic mmap ENOMEM. Per-slab + sequential,
so each payload's slab sees the free remaining after the prior ones.
Unit-tested (capacity check: tiny fits, over-capacity fails loud). Pre-existing 2b L2-pooling code (not L3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The collector was created only in _maybe_init_cp_l3 (enable_cp_l3 path), so a pure-L2-pooling
deployment (no L3) got no L2 metrics -- exactly the config under perf investigation. Move creation
into _cp_maybe_collect_metrics: lazily create on first emit, gated on enable_metrics + the shared-L2
allocator existing (L2 pooling on). collect() already handles l3_stats=None, so it reports L2
occupancy/commit/evict with or without L3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under CP, enable_storage=False, so StorageMetricsCollector (prefetch/backup tokens + bandwidth) never
emits, and L2 pool usage was debug-log-only (_cp_maybe_log_lane_stats) while L3 was invisible. Add a
CpHiCacheMetricsCollector covering both, gated on enable_metrics alone, per-rank via the tp_rank label.
- CpL3Store.stats(): per-payload slot occupancy (used,total) + cumulative counters (spill_pages,
reload_pages, gc_reclaimed, write_errors -- incremented on the bg threads) + instantaneous inflight
(spill/reload) + the four bg queue depths. Counters are monotonic (NOT reset on clear()).
- CpHiCacheMetricsCollector (metrics_collector.py): gauges for instantaneous state
(sglang:cp_l2_used/total_pages, cp_l3_used/total_slots, cp_l3_inflight_*, cp_l3_queue_depth) +
counters for cumulative work (cp_l2_objects_committed/evicted_total, cp_l3_spill/reload_pages_total,
cp_l3_gc_reclaimed_total, cp_l3_write_errors_total). Counters fed via a monotonic .inc(delta) off the
store's cumulative totals.
- hiradix: create the collector in _maybe_init_cp_l3 (once, per-rank labels from cache_controller);
push periodically (~1s, per-rank wall-clock -- metrics need no rank-uniformity) from
check_hicache_events via _cp_maybe_collect_metrics, reading allocator.occupancy_by_payload()/stats()
+ store.stats(). Off the data path.
store.stats() unit-tested (occupancy + spill/reload counters; 10 store tests green). The collector +
wiring are correct-by-pattern (mirror StorageMetricsCollector) and validate on the live server (3.4;
prometheus_client not in the unit env). Directly useful for 3.4: watch L2/L3 fill + GC + reload hits
under cachebench.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces §9.3's object-level rank-uniform eviction with per-rank, page-level LRU GC running ON THE
WRITE THREAD (the sole slot-pool owner -> no lock, no second owner). Genuinely communication-free:
each (hash,payload) is written/read/deleted by exactly its one owner (owner = i%cp_size, and the
prefix-chained content hash makes one-hash = one-position = one-owner), so the shared LMDB's own
consistency substitutes for collectives within L3 (design §9.5/§9.6). GC touches only L3, adds zero
collectives; the spill/reload MINs that remain are all the L2 bridge.
cp_l3_store.py:
- Per-payload lazy-deletion min-heap `_gc_heap` of (last_access, seq, hash) + authoritative
`_gc_current{hash->(last_access,slot)}`. Add on write, touch via `_touch_q`, reclaim coldest.
- The write loop, each iteration: drain touches -> _gc_collect (reclaim coldest of any pool over the
0.90 start watermark down to 0.85, bounded budget) BEFORE the next write, so spill never hits a full
pool (no evict burst on the critical path). Continuous + proactive, symmetric with the spill.
- Reclaim deletes the index entry (durable) BEFORE freeing the slot, so a racing reload reads None or a
CRC-mismatched reused slot -> miss -> recompute (fail-soft); no lock / in-flight-exclusion needed.
- submit_touch (scheduler->write thread); clear() resets the GC structures.
hiradix_cache.py: wire the reload `touch` (the 3.2 gap -- hit_count/last_access never updated): at
_cp_l3_admit_reload, bump the L3 last_access of THIS rank's owned reloaded pages (ALL pages, not just
the tail, so a prefix ages uniformly). _cp_l3_insert_reloaded_node now returns the node (its
last_access seeds the touch).
Sharing handled optimally for free (page = one entry, last_access = max over touchers; prefix pages
age together; shared-hot survives) -- no object-record table, no refcount, no content-keying. 56 L3
unit tests green (new: GC reclaims coldest-to-watermark + no slot leak; touch keeps a page warm).
Cold-rebuild (scan slab headers + LMDB) is designed (§9.6) but deferred to 3.4 (crash-recovery).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a radix miss, reload the evicted-but-L3-durable suffix from disk into L2 + insert a fresh
radix node, holding the triggering request until then (mirrors the storage-prefetch hold, which
is disabled under CP). The reloaded prefix re-enters the radix as a normal L2 node, so the
existing load_back serves L2->L1 -- no new device path.
Design (docs_internal/cp_hicache_l3_phase3_impl_design.md §9, refined for minimal sync):
- ENTRY (scheduler _prefetch_kvcache, enable_cp_l3 branch): mark a miss-with-suffix as a reload
candidate (rank-uniform: requests are broadcast). cp_l3_reload_lookup computes the suffix's
full-page content hashes -- the SAME SHA chain spill wrote (compute_node_hash_values).
- RESERVE + ADMIT fold into the existing per-tick _drain_l3_control_queues MINs -- ZERO new
collectives: the candidates' per-rank exists_prefix counts ride the qsize MIN vector -> a
rank-uniform agreed count C (this reconciles the async-LMDB read-skew); reload-ack oks ride the
durable ok-AND. Reserve is then collective-free (deterministic evict-to-fit mirroring
_reserve_write_cp_shared_l2_evict_to_fit); the request waits non-blocking in waiting_queue
(check_cp_l3_reload_progress skip) and is admitted + re-matched the SAME tick the reload lands
(check_hicache_events runs before batch formation). CP-aware insert only on a still-clean attach
(else abort+recompute). Capped (cp_l3_reload_max_inflight) + content-key deduped (piggyback).
EAGLE/bigram (opus-studied, PROVABLY correct -- not deferred): the radix key is bigram-converted
over the whole request, and convert_to_bigram_key(fill_ids[M:]) == bigrams(fill_ids)[M:] exactly
(the boundary bigram belongs to the matched prefix), so the bigram-converted suffix hashes
reproduce node.hash_value. Verified vs source + a new slice-identity regression test.
Opus review (FIX-THEN-SHIP, no CRITICAL; the 4 crash surfaces -- MIN-vector shape, read-skew,
attach-anchor, placement-digest -- traced clean) + independently re-verified; fixes folded:
- M1: cp_l3_release_request clears reload state on request abort/timeout/preempt (no leak).
- H1: rank-uniform per-op TTL bound releases a held request if its reload op never acks (the
default SGLANG_REQ_WAITING_TIMEOUT is off); the reservation frees safely at the (late) ack.
- L1: fail-soft anchor guard at insert (re-derive the first suffix hash from lhn's live hash).
- empty-waiters reload aborts (frees L2) instead of inserting an unrequested node.
Imports hoisted to top-level (get_hash_str -- hicache_storage is a leaf module, no cycle;
convert_to_bigram_key already top-level), no inline imports. py_compile clean; 54 L3 unit tests
green. Reload triggering (submit_reload/exists_prefix/free_object) now wired (was write-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first 3.1 (d08ee9f8d) spilled the COLDEST resident objects and held the
protect_host eviction-pin through the entire slow disk write (released only at the
durable ack). Since the coldest objects ARE eviction's victims, a B300 cachebench
wire test wedged: evict-to-fit found everything pinned (evicted=0) ->
host_reservation_failed flood -> caching broke.
Redesign (docs_internal/cp_hicache_l3_phase3_impl_design.md §R3):
- cp_l3_store: split the single spill thread into a GATHER -> WRITE pipeline. Gather
copies each owned page off the pinned slab into a slab-independent anonymous-mmap
staging buffer (fast RAM memcpy) and acks GATHER; a separate write thread does the
O_DIRECT write + fdatasync + durable LMDB put off the staged copy and acks DURABLE.
The eviction-pin releases at the GATHER ack, so the object is evictable right after
the RAM copy; the disk write completes off the staged copy even if L2 reuses the
pages. Slab-full frees its partial slot allocation (no orphan) + acks ok=False.
- hiradix: continuous PROACTIVE spill — enqueue each just-committed object to a FIFO
deque at the replicated commit frontier (_commit_pending_backup) AND at radix splits
(the freshly-committed parent half), drained in commit order (not coldest). The deque
is capped (rank-uniform) so a disk stall can't OOM it. Spilling HOT just-committed
objects (disjoint from eviction's COLD victims) means the cold tail is already
L3-durable when evicted -> eviction just drops it, never contends for the pin.
- 3-element CP-group MIN drain [gather, durable, reload]: gather-MIN -> release_host;
durable-MIN -> l3_durable, gated by an ok-AND (a second MIN over per-op ok bits) so a
write failure on any rank means NO rank marks it durable (rank-uniform durability,
placement_digest stays green).
opus-reviewed (FIX-THEN-SHIP) + independently re-verified; review fixes folded:
split-enqueue (HIGH), deque cap for the disk-stall OOM (MED), fail-soft-on-write-failure
documented (HIGH; deliberately NOT re-enqueued — re-gather would churn under slab-full,
recompute is correctness-safe), rate-limited error logs (MED/LOW). 54 L3 unit tests
green (new: 2-phase gather-before-durable ordering, slab-full fail-soft + no slot leak).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spill is decoupled from eviction (approach D, user-confirmed): a rank-synced background maintainer at
check_hicache_events COPIES the coldest resident committed objects (replicated SLRU order, not-yet-in-L3,
unpinned) to L3 off the critical path. The object stays L2-resident; protect_host pins the LIVE node during
the async bg gather (no freed-range hazard). owned_pages owner = object-local i%cp_size (verified vs
_shared_l2_current_page_owners); per-payload base_page from object_ranges; logical pages only. Bounded +
backpressured (cp_l3_spill_max_inflight). Replicated selection -> lockstep object-granular acks. Spill-ack
(in _drain_l3_control_queues): release_host + node.l3_durable=True (O(1) skip; memoized from exists_prefix
for reloaded nodes). Eviction unchanged (drops L2; L3 copy survives -> reload in 3.2; not-yet-spilled =
recompute, fail-soft). Compiles; default-off. Live validation via flag-on smoke (branch sync).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_maybe_init_cp_l3: build the 3 CpSharedL2SlabAccessor (target_kv always; draft_kv if draft pool;
index_k if NSA) from the LIVE host pool slabs (verified attrs: host.allocator.mapping.mmap,
layer_num/page_num/kv_cache_dim/dtype.itemsize; index_host_tensor_allocator + indexer attrs), then
CpL3Store.from_config + connect (PLP gate). Single-slab only (default --cp-shared-l2-slab-size-gb 0);
fail loud on the multi-slab group case (future extension, not a silent stub).
_drain_l3_control_queues: per-tick CP-cpu-group MIN over the 2 ack-qsizes + drain MIN of each, idle
early-return on the replicated has_inflight (R1 CRIT-3/4) — wired in check_hicache_events (gated on
enable_cp_l3, NOT the enable_storage drain). reset() -> cp_l3_store.clear(); shutdown() -> close().
Default-off + behavior-neutral. Compiles. 3.0 CODE complete; flag-on smoke gated on cluster branch sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
server_args: --enable-cp-l3 + --cp-l3-config fields/args. Validation (R1 HIGH-2): do NOT re-gate
the 6 CP storage forbids (they gate on hicache_storage_backend and correctly never fire); instead
assert enable_cp_l3 is mutually exclusive with hicache_storage_backend, requires
enable_cp_shared_physical_l2_hicache, and requires cp_l3_config.
hiradix: self.enable_cp_l3 (CP-only) + cp_l3_store placeholder; compute_node_hash_values now also
fires under enable_cp_l3 (the per-page content hash is L3's key) — clean-boot only (R1 HIGH-3).
Default-off + behavior-neutral; store construction + drain follow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ties config -> per-rank disk slabs + slot pools + shared LMDB index + slab accessors. Background
spill/reload threads (off the scheduler tick, storage-template shape); object-granular acks (one per
object after all owned pages durable; zero-owned ranks ack in lockstep) via ack queues + has_inflight,
so the caller's CP-cpu-group MIN-drain frees the same objects on every rank. Spill = gather->O_DIRECT
write->durable index (data->fdatasync->index-commit ordering) with content-hash dedup; reload =
index lookup->read_into->scatter (verify-on-read). free_object frees owned slots+index (3.3 eviction
consumes); clear() is the flush_cache hook (stop/drain/reset/restart). from_config splits the disk
budget across ranks-on-disk x payloads (equal slot count). PLP gate at connect(). Model B: store is
content-addressed + rank-local I/O; only the LMDB is shared. End-to-end test (spill/exists/evict/reload
byte-exact/dedup/free/clear) 5/5 (venv lmdb). + accessor n_layers/page_num accessors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CpSharedL2SlabAccessor (HIGH-1): memcpy-gather a page's N_layers strided slab slices into a
contiguous buffer (spill) + scatter a contiguous blob back into the strided slices (reload) —
the cheap RAM bridge that lets host stay layer_page_first (inference perf) while disk is
page-contiguous. Generic CpL3SlabLayout(n_layers, page_num, slice_bytes) with verified-from-source
factories: for_mla (layer_num,page_num,page_size,kv_cache_dim,itemsize -> 2.74 MiB/page) and
for_index (n_active_layers,indexer_page_num,indexer_page_stride -> 0.17/0.63 MiB/page). gather_into/
scatter_from take an offset so the spill writes straight after the 64B slot header (zero extra copy).
Pure (mmap+ints), no torch. 8/8 tests. Canonical layer 0..N-1 order shared by spill+reload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Preallocated disk-slab file per (payload_kind, drive); a page = one contiguous blob
(64B header + gathered payload + pad) in a 4K-aligned slot. O_DIRECT single-transfer
write/read (measured best vs vectored), buffered+fdatasync fallback for FS without O_DIRECT.
write_slot/read_slot convenience + write_aligned/read_into zero-copy (the QD>1 spill/reload
path gathers/scatters straight into the aligned slot buffer). verify-on-read: absent/torn/
foreign slot -> None (a MISS to recompute + reclaim), never a crash. posix_fallocate preallocate.
9/9 unit tests (real temp FS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CpL3Config (TOML/JSON): per-machine L3 disks (paths + budgets), backend, shared index
location/size, and the rank->disk mapping. disk_for_rank balances M CP ranks over N disks
(r % N; ranks share a disk gracefully — throughput is drive-count-bound) or honors an
explicit map; fail-loud on length/range mismatch. probe_disk_plp: O_DIRECT write+fdatasync
latency heuristic for the durability gate (non-PLP disk must fail loud when require_plp).
No hardcoded paths. 11/11 unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3.0 (L3 disk durable floor) foundational primitives, pure + unit-tested:
- 64B self-describing blob header (magic/version/payload-kind/content-hash/CRC),
verified on every read (verify_blob) -> torn/bit-rot slots become a MISS, not garbage.
- slot_bytes_for_page: header+payload rounded to the 4K O_DIRECT boundary.
- CpL3SlotPool: O(1) free-list over fixed-size per-payload disk page-slots (the L3
analog of CpSharedL2PageAllocator's free list), fail-loud on double-free/OOB, with
reset() for the flush_cache hook. NOT a ring buffer (eviction frees arbitrary slots).
Rank-local (a rank stores only its owned pages; L3 eviction selection rides the shared
LMDB replicated clock). 12/12 unit tests. Design: docs_internal/cp_hicache_l3_phase3_impl_design.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the interim ack_queue_len==0 band-aid with a structural fix so the perf skip is
SAFE rather than removed.
Root cause: the radix attach/defer/prune/split probes read `_node_host_write_pending`
(membership in ongoing_write_through/pending_host_backups, which are drained ASYNCHRONOUSLY
at per-rank ack completion), and `_cp_subtree_has_unprunable_state` also read the rank-local
`lock_ref`. So per-rank ack-drain timing leaked into the match_prefix truncation ->
cache_protected_len -> logical_len/len(value) -> the attach predicate. On some ranks the
prepared backup attached, on others it was dropped to the synchronous write_backup catch-up
(catch_up_all_layers draft=True), whose ack-append is per-rank. ack_write_queue then diverged
across CP ranks, so writing_check's ack_queue_len==0 early-return diverged: a strict subset
entered the writing_check_min all_reduce while the rest advanced to recv_requests' broadcast
on the same tp_cpu_group (8 ranks; dp_size==1 resets enable_dp_attention to False) -> NCCL
deadlock (prod hang: node_id=50713, all 8 GPUs 0% util).
Fix: introduce TreeNode.cp_backup_pending, a rank-replicated marker set at the rank-replicated
prepared-backup attach (_attach_prepared_cp_backup) and the guarded write_backup registration,
cleared only at the MIN-committed commit (_commit_pending_backup) / rollback. Rewire the CP
radix probes (_cp_node_split_still_pending, _split_node guard, _cp_subtree_has_unprunable_state,
_inc_hit_count) to read this marker instead of the drain-timed dicts, and DROP the lock_ref read
in the prune probe (its only cross-rank skew is the backup lock, now covered by the marker).
_node_host_write_pending stays the drain-timed accessor used only by writing_check / eviction /
the visibility check (_node_host_write_ready). With the radix-path reads replicated by
construction, len(value)==logical_len on every rank -> the prepared backup always attaches (or
rolls back) symmetrically -> the asymmetric catch-up never fires -> ack_write_queue is symmetric
-> the existing ack_queue_len==0 skip is all-or-none (no per-tick no-op MIN, best perf).
Correctness: the marker is cleared only at/after _commit_pending_backup, which runs only under
the MIN frontier, so the MIN remains the SOLE host-visibility gate (no KV-corruption class
reopened); the marker is strictly more conservative than the old dicts (stays True until commit).
Fail-fasts: double-attach raises; _split_node propagates the marker. hi_mamba unchanged (gates
on ongoing only, no async ack skip). Single-layer-draft (c3fc3ff752) preserved: the design
touches only the attach decision; the draft notifier already fires post-MoE.
Adds test_cp_hicache_symmetric_attach.py: the probes read the replicated marker not the dicts,
the prune probe ignores lock_ref, and the marker lifecycle (attach->commit->rollback) + the
double-attach guard. All pass on the fix; all fail on the pre-fix code.
Design: docs_internal/cp_hicache_symmetric_attach_design.md. Pending: ETE (GSM8K + no-hang
flood + TTFT) on the dev-cu13 container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 876a98177cd749b5a63623ba2ae8b868289b8e59)
With DeepEP the sparse-MoE layer keeps tokens SCATTERED (DeepEP dispatch/combine
routes tokens to expert-owning ranks), so the NSA layer communicator never
materializes the FULL hidden states. Decoupling CP onto the standard/allgather
a2a (flashinfer_trtllm, moe_a2a_backend=none) flips mlp_mode to FULL, which makes
NSACPLayerCommunicator gather hidden states across CP ranks before MoE and scatter
back after.
The gather used a raw even attn_cp_all_gather_into_tensor into get_local_dp_buffer(),
and the scatter used tensor_split(cp_size) + reduce_scatter. Both require equal
per-rank shards. Under --enable-nsa-prefill-cp-shared-kv the per-rank shards are
UNEVEN: the page-aligned in-seq split page-rounds each request's 2*cp_size segments,
so per_rank_actual_token differs across ranks. Result: "output tensor size must be
equal to world_size times input tensor size" in the CP all-gather.
Fix: make the round-trip pad-aware, mirroring the NSA index/KV gather idiom that
already handles uneven CP shards via max_rank_len.
- Gather: _cp_attn_tp_all_gather_padded_tensor pads this rank's shard to
max_rank_len and all-gathers into a [max_rank_len * cp_size, H] rank-major
buffer (allocates its own buffer; get_local_dp_buffer()'s length can be smaller
than max_rank_len*cp_size once page-padding inflates the per-rank max).
- Scatter: tensor_split(cp_size) now yields equal max_rank_len chunks (canonical
in-place reduce-scatter layout); reduce-scatter sum-combines this rank's EP-local
MoE partials, then slice to per_rank_actual_token to drop the padding rows,
realigning with the SCATTERED residual.
Strict generalization: when shards are even (non-shared-KV) the padding is zero and
this reduces to the previous behavior. Gated by nsa_use_prefill_cp, so decode/non-CP
and the DeepEP baseline (mlp_mode SCATTERED) are untouched. No global_num_tokens
2*cp_size padding needed: max(per_rank_actual_token)*cp_size == ceil_align(total,
cp_size), which the existing cp_size alignment already provides.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CP draft shared-KV target prefill only needs the CP-local draft hidden side channel until draft prefill consumes it. The reused ModelWorkerBatch was left with capture_draft_hidden_states enabled, and LogitsProcessorOutput retained the extra draft_hidden_states tensor past that point. Restore the capture flags after target prefill, disable draft-hidden capture for draft prefill, and drop the consumed reference.
Constraint: Large CP prefill batches are memory-sensitive; an extra hidden tensor held across the speculative step materially increases peak memory.
Rejected: Leave capture_draft_hidden_states sticky across draft prefill | it can capture or retain hidden state outside the target-side CP-local side channel.
Confidence: high
Scope-risk: narrow
Directive: Keep CP-local draft hidden as a target-prefill-only side channel unless the draft worker explicitly owns a new lifetime contract.
Tested: g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_paged_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/speculative/test_eagle_worker_v2_cp_hidden.py
Not-tested: full speculative integration/E2E workload.
(cherry picked from commit f31ef2293ce0cdda2359a5f4713996d78b47f9df)
