ROOT PROBLEM (opus-diagnosed): the NextN/MTP draft MoE is a normal DeepSeek-V3 fp4 MoE, identical in
kind to the main model's — but our fork forced the speculative MoE backend to "auto" (+ a blanket
assert forbidding trtllm-as-speculative), so the draft built under speculative_moe_backend_context →
MOE_RUNNER_BACKEND=AUTO → plain FusedMoE → the non-flashinfer process_weights 'else' branch that
doesn't reconcile EP scale sharding → w13_input_scale(global 256) × w13_weight_scale_2(local 64) crash.
The trtllm-as-speculative ban is over-broad: the Renormalize-only routing constraint is for the BF16
trtllm MoE; the FP4 trtllm kernel supports the DeepSeek routing the NextN carries (FlashInferFP4MoE.
forward_impl). So the elegant fix is to let the draft run the SAME flashinfer_trtllm path as main when
the NextN is fp4 — scoped via the checkpoint's safetensors index (ground truth: do the MTP experts
ship fp4 scales). DeepSeek-R1/V3-FP4 ship a bf16 MTP -> still drop to bf16 + keep the non-trtllm path.
- weight_utils: add shared nextn_moe_is_fp4_quantized_in_checkpoint + load_safetensors_index_weight_map.
- deepseek_nextn: delegate the keep/drop probe to the shared helper (one source of truth).
- server_args._handle_speculative_decoding: for an fp4 NextN, default the speculative backend to the
main trtllm backend (instead of "auto"); narrow the assert to allow trtllm only for an fp4 MTP.
- modelopt_quant: REVERT the else-branch input-scale slice band-aid (de726b4e7e) — no longer hit; the
fp4 draft now takes the trtllm .max() scalar branch, same as main.
Bug B (k_rope 256-vs-64 in the trtllm NSA *prefill* path under CP) is independent and still pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The NextN/EAGLE draft runs under speculative_moe_backend_context -> MOE_RUNNER_BACKEND=AUTO (trtllm is
forbidden as a speculative backend), so its NVFP4 MoE takes the plain-FusedMoE 'else' branch of
process_weights_after_loading. That branch left w13_input_scale/w2_input_scale GLOBAL (num_experts=256,
they are tagged _sglang_require_global_experts at load) while w13_weight_scale_2 is EP-local (64) ->
_compute_gemm1_alphas did 256*64 -> RuntimeError. trtllm/cutlass collapse to a scalar and cutedsl already
slices; the else/triton path was a latent EP>1 NVFP4 bug. Slice the global input scales to this rank's
local experts (mirrors the cutedsl _slice_scale). No-op for moe_ep_size==1; main model unaffected
(never takes this branch). opus-diagnosed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runner-backend heuristic is fundamentally the wrong signal: it reads AUTO in the EAGLE draft
worker (verified live), and an fp4-capable runner does not imply an fp4 MTP — DeepSeek-V3-0324-FP4
runs --moe-runner-backend flashinfer_trtllm yet ships a bf16 MTP (CI test_deepseek_v3_fp4_mtp_small).
is_layer_excluded is also disqualified: R1/V3 leave the MTP unquantized-but-absent-from-`ignore`.
Complete fix (opus-designed): inspect the checkpoint's model.safetensors.index.json and keep fp4 iff
the NextN/MTP experts (model.layers.{num_hidden_layers}.mlp.experts.*weight_scale*) ship fp4 scales —
the ground truth. GLM-5.2-NVFP4 has them (verified: 1536 layer-78 scale keys) -> keep; R1/V3 don't
-> drop. Unreadable index -> drop (PR #7376 default, bf16 MTP fallback, never crashes). Removed the
backend heuristic + debug log + unused get_moe_runner_backend import. Resolves both the 6144-vs-3072
draft crash AND the flood of "model.decoder.mlp.experts.*_scale not found in params_dict" (same root
cause: dropping fp4 left the unquant MTP with no scale params for the checkpoint's fp4 scales).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The NextN/MTP draft is built by the EAGLE draft worker, where get_moe_runner_backend() didn't resolve
to flashinfer_trtllm, so the fp4 drop fired anyway -> 6144-vs-3072 crash in draft load. Check both the
main and speculative runner backends (spec is AUTO when --speculative-moe-runner-backend unset); keep
fp4 if either is a flashinfer fp4 backend. Temporary [NextN-fp4] warning logs the resolved backends.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The team-custom FlashInferFP4MoE.forward_impl read gemm{1,2}_{weights,scales}_fp4_shuffled, but the
HEAD-advanced align_fp4_moe_weights_for_flashinfer_trtllm now stores into w13_weight/w2_weight/
w13_weight_scale/w2_weight_scale (weights uint8, scales already fp8). Rename the 4 reads (Option a:
preserves the team forward incl. its DeepSeekV3 fp32 router_logits cast — GLM-5.2 uses DeepSeekV3
routing, so option-b delegation that drops the cast was avoided). g1_scale_c/g1_alphas/g2_alphas/
w13_input_scale_quant already match HEAD writes (opus-verified 1:1 format/semantics, no silent-garbage).
Cannot wholesale-port layer.py/ep_moe (team EP/CP infra). Watch at launch: draft accept-len + sane output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ported flashinfer_trtllm.py (Stage 1) calls prepare_static_weights_for_trtllm_fp4_moe(is_gated=...),
but utils.py (which DEFINES it) was not ported → old signature → TypeError at process_weights.
utils.py has 0 local edits; wholesale-replace to HEAD. New fn is gated-aware (gemm1_rows), passes
is_gated_act_gemm to flashinfer _maybe_get_cached_w3_w1_permute_indices (verified present on 0.6.12).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GLM-5.2-NVFP4's NextN (MTP) layer-78 routed experts are fp4, but deepseek_nextn unconditionally
dropped modelopt_fp4 for NextN (a DeepSeek-R1-FP4 assumption: its MTP is unquantized). Two fixes
(opus-designed, full back-compat matrix):
A) Drop fp4 for NextN only when the MoE runner can't consume it. Keep fp4 for flashinfer
trtllm/trtllm_routed/cutedsl/cutlass (extends wxiwnd's helper to include our trtllm runner).
R1-FP4/V3-FP4 (non-flashinfer-fp4 or unquantized MTP) still drop → bf16, no regression.
B) The NextN hf_to_sglang_mapper was hardcoded `model.layers.61 -> model.decoder` (R1's MTP idx).
For GLM (MTP=layer 78) this never remapped the layer-78 self_attn/indexer exclude patterns to
the internal model.decoder prefix → NextN attention wrongly quantized fp4 → crash. Worse, keeping
the hardcoded 61 would mis-remap GLM's *real* layer 61. Fix: empty the class mapper and inject
`model.layers.{num_hidden_layers} -> model.decoder` per-config in loader.py (gated on
num_nextn_predict_layers; dataclasses.replace, no class mutation), generalizing to any MTP index.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NSA-prefill CP in-seq-split no longer hard-forces moe_a2a_backend=deepep. Only default to deepep
when the user left both moe_a2a_backend=none AND moe_runner_backend=auto; with an explicit runner
(e.g. flashinfer_trtllm) keep a2a on the standard/allgather path, validated to {deepep,flashinfer,none}.
Adapted from wxiwnd@b300-combact-fix 317acf6c99 (reconciled to our HEAD). Also relax the tp_size==8
assert to single-node tp_size in {4,8} so the GLM-5.2-NVFP4 tp4/CP4 PD target can start; CP code is
parametric in cp_size and the tp%cp / tp%(dp*cp) divisibility asserts still guard validity.
GLM-5.2 (GlmMoeDsaForCausalLM = DeepSeek v3.2 arch) inherits DeepseekV2DecoderLayer, which already
selects NSACPLayerCommunicator under CP -> no GLM-side communicator change needed (decouple stage B
N/A; Glm4MoeDecoderLayer is the unrelated GLM4 arch). Runtime FULL<->SCATTERED verify pending launch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the flashinfer port ("port全"): ADD flashinfer_cutedsl.py (C2, deepep-paired NVFP4 MoE
runner; selectable via --moe-runner-backend flashinfer_cutedsl, already a MoeRunnerBackend enum) and
REPLACE flashinfer_comm_fusion.py to HEAD (C3, allreduce fusion). Both pull self-contained infra
modules target lacked: ADD model_executor/cuda_graph_config.py (cuda_graph_fully_disabled, 0 sglang
top-level imports) and runtime_context.py (get_parallel, lazy imports). comm_fusion consumers
(communicator.py is_flashinfer_allreduce_unavailable, layernorm.py flashinfer_allreduce_residual_rmsnorm)
verified present in HEAD.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two latent CP shared-KV L2 correctness fixes, landed BEFORE L3 (not in an L3
commit). Surfaced by a per-CP8-instance (CP8DP2EP16) scoping review.
PREREQ-1 (CP-group scoping). The B1 commit/evict consensus collectives
(writing_check ReduceOp.MIN, placement_digest MIN/MAX + its tp_world_size<=1
entry guard, drain_storage_control_queues, the evict/prefetch MINs, the flush
barrier) AND cache_controller.prefetch_tp_group all derive from
self.tp_group/self.tp_world_size, which was params.tp_cache_group. tp_cache_group
equals the CP group ONLY at dp_size=1 (enable_dp_attention False -> tp_cpu_group,
and attn_cp_size==tp_size -> _ATTN_CP==_TP) -- the sole reason B1 works today.
Under CP8DP2 (DP attention, attn_tp_size=1) tp_cache_group is the size-1 attn-TP
group, so every `tp_world_size>1` collective silently no-ops per rank and the
placement assert self-disables -> divergent placement -> shared-slab corruption.
Fix: for CP hicache (cp_size>1) scope self.tp_group to the CP cpu group
(get_attention_cp_group().cpu_group -- already used for the slab-handle
broadcast) + self.tp_world_size to its size. A no-op handle change at dp_size=1
(same group object); the intended fix at CP8DP2. The single init-point change
propagates to every consensus collective + un-gates prefetch_tp_group + re-enables
the placement assert.
PREREQ-2 (flush reset). HiRadixCache.reset() cleared the radix tree + host pool
but never reset CpSharedL2PageAllocator -> stale free list/ranges/committed after
flush_cache (leak; the shared pool was never reclaimed). Added
CpSharedL2PageAllocator.reset() (rebuild the per-slab free list all-free, drop
ranges + committed, restore the freshly-built placement_digest) called from
cache_controller.reset() after the ack queues are cleared. Safe: flush_cache is
idle-gated (no in-flight backup/reserve). This is also L3's clear hookpoint.
Validation: new test_reset_restores_freshly_constructed_all_free_state + 89/89
pool suite (torch-2.11 container) + import smoke. PREREQ-1 is a no-op at dp_size=1
(live no-regression confirmed on the next prefill restart); CP8DP2 correctness is
by construction (CP-group membership verified) pending a 2-machine run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Target-model skip-topk layers reuse the previous active layer's top-k indices and should not run local indexer modules. Centralize the layer-needs-indexer decision, skip constructing indexers on shared target layers, and skip their checkpoint tensors during load while keeping nextn/draft layers conservative for state safety.
Constraint: index skip should reduce GPU memory in both prefill and decode without changing top-k propagation semantics
Constraint: nextn/draft layers report shared top-k behavior but still need local indexer state safety
Rejected: Loader-only filtering | parameters are already allocated during model construction
Rejected: Dummy indexer modules for skipped layers | preserves most of the memory cost this change removes
Confidence: high
Scope-risk: moderate
Directive: Do not reintroduce indexer execution on skip_topk target layers without proving prev_topk propagation and weight residency semantics
Tested: remote g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/speculative/test_spec_utils.py test/registered/unit/configs/test_nsa_index_layers.py test/registered/unit/models/test_deepseek_index_skip_weight_loading.py -> 19 passed
Tested: remote g0034 cjy-glm5-new py_compile for modified runtime files
Not-tested: full GLM5 model restart memory delta measurement
EAGLE verification can pass torch scalar tensors through the speculative tree traversal before calling grammar backends. xgrammar/tvm_ffi requires Python int token ids, so normalize traversal indices and draft token ids at the boundary while leaving tensor storage unchanged.
Constraint: xgrammar/tvm_ffi rejects torch scalar tensors for GrammarMatcher.accept_token
Rejected: Coerce tokens inside every grammar backend | the invalid value originates in speculative traversal and should be fixed before backend dispatch
Confidence: high
Scope-risk: narrow
Directive: Keep grammar backend calls scalar-Python typed; do not pass torch scalar tensors through accept_token
Tested: remote g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/speculative/test_spec_utils.py test/registered/unit/configs/test_nsa_index_layers.py test/registered/unit/models/test_deepseek_index_skip_weight_loading.py -> 19 passed
Tested: remote g0034 cjy-glm5-new py_compile for modified runtime files
Not-tested: live decode replay after this commit
Tier-2-GC follow-on (design §9.12 flagged vestige). The metadata's `committed`
bool and `committed_payload_layers` dict were write-only residue of the option-A
quorum the GC removed: set at construction, validated in __post_init__, copied in
split() -- but their only content/True writer was the deleted
_commit_cp_shared_l2_layers_batched. Verified tree-wide there is NO live reader
(`committed` appears only in split's `committed=self.committed` copy;
`committed_payload_layers` only in __post_init__ validation + split copy +
construction). Commit consensus under B1 is the allocator's `_committed_objects`
(MIN frontier), so `metadata.committed` was a redundant second source of truth.
CORRECTION to the original flag: `required_payloads` is NOT a vestige -- it is
LIVE (read by cache_controller.load_cp_shared_l2, the evict path hiradix:2748,
and the readback payload planning). KEPT.
Removed: the 2 dataclass fields + their __post_init__ validation + the two split()
child-copies + the cache_controller reserve-construction kwargs + 4 test
constructions/assertions. No behavior change (fields were never read). Pool suite
88/88 (syh-dev-new/torch-2.11 env) + import smoke confirm the metadata dataclass is
now {logical_len, padded_len, page_size, object_ranges, required_payloads, object_key}.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When --enable-cp-shared-physical-l2-hicache is on, the lane-stats occ/used_tokens
read 0 (they derive from _cp_counts, which the accounting skips for shared-L2
metadata) -- the per-rank gauge is meaningless. Repoint occ/used/cap at the
shared pool's per-payload-slab fill (target_kv/draft_kv/index_k) via the new
CpSharedL2PageAllocator.occupancy_by_payload(); add occ_by={payload,rank} +
occ_labels so the line is self-describing. The real cp_size (owner round-robin
period for the hot-prefix skew) is now computed independently of the occ-vector
length. Per-rank (flag-off) path unchanged. Read-only; rank-0; next-restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CP_HICACHE_LANE_STATS occ/used_tokens are derived from _cp_counts, which the
accounting explicitly skips for shared-L2 metadata (_cp_account_*: `not
_is_cp_shared_l2_metadata`), so they read 0 with l2 pooling on regardless of
actual fill -- wrong gauge for the shared pool. Add cache_controller
.cp_shared_l2_cache_report() (pages_used / pages_capacity / objects_committed /
objects_aborted / objects_evicted + load_hits/misses from the replicated
CpSharedL2PageAllocator stats) to the lane-stats line as `shared_l2_pool=...`
when the allocator is constructed, so fill->evict->load-back is observable.
Read-only; rank-0 only; takes effect on next restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flag-on 2b.3 bring-up (--enable-cp-shared-physical-l2-hicache, real GLM-5.1
PD traffic) surfaced two partial-adopt gaps from the piecemeal 2b.0/2b.1b ports
from l2_pooling — a caller brought in without its callee/declaration. Neither
was caught by the pool unit tests or import smoke (they don't exercise the
device-allocator load path or the flag-on capacity-snapshot path); only live
flag-on traffic hit them.
1. environ.py: declare SGLANG_CP_HICACHE_VERIFY_SNAPSHOT (EnvBool, default False).
hiradix _cp_host_capacity_snapshot (:1477) reads it to gate a fail-loud
running-count drift verifier (_cp_walk_capacity_counts vs _cp_counts), but the
env field was never declared -> AttributeError crash on the first flag-on
scheduler tick. The verifier IS fully implemented (it's the count-machinery
analog of the placement_digest assert) -- keep it, default-off (expensive
full-walk; on in bring-up/CI).
2. allocator.py: restore alloc_pages_for_shared_l2_load (thin wrapper over the
adopted alloc_pages_with_owners) on the CP shared paged allocator. 2b.0 ported
the callee but dropped this wrapper that cache_controller.load_cp_shared_l2
calls -> RuntimeError on every L2->L1 cache-hit reload until restored.
Validated live: prefill boots with the flag, shared-L2 slab constructs over
hugetlbfs on all 8 ranks, write-through + load-back + cache hits + coherent
output all work, placement_digest assert green (no divergence), no crash.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the 2b.1b shared-L2 capacity-miss SKIP with a deterministic
collective-free evict-to-fit, and deletes the #5 deadlock. This is the gate that
makes --enable-cp-shared-physical-l2-hicache safe in a real run.
evict-to-fit (_reserve_write_cp_shared_l2_evict_to_fit, hiradix): on a shared-L2
reserve capacity miss -- which raises identically on every CP rank over the
replicated free list -- snapshot the plannable shared-L2 host leaves (excluding
the node being backed up) in the replicated SLRU order (_cp_host_evict_key =
Phase-1 logical clock + node.id total-order tiebreak), release the coldest one at
a time (each a replicated free-list mutation via the proven
_evict_cp_host_for_write_admission teardown -> evict_cp_host -> allocator.release)
and retry the pooled reserve after each, stopping at the first fit.
Fragmentation-correct (free coalesces in _return_range). Identical victim
set + order + deterministic reserve => every rank evicts the same victims and
stops at the same point (design Thm 1), so NO collective is needed. The snapshot
is one-level by design (a parent promoted to a leaf mid-loop is not re-pushed;
deep cascades skip-and-retry next tick -- missed identically on all ranks).
Exhaustion -> loud rate-limited skip (transient: pinned/in-flight objects hold
the pool), never a hang.
#5 deadlock DELETED: stripped the synchronize_across_ranks all_reduce machinery
(the `while len(heap) and not all_ranks_done()` ReduceOp.MIN host_evict_done_min
closure + the param) from _evict_host_for_physical_slots -- never reached (no
caller passed True) and the headline CP-deadlock shape. evict_host(num_tokens)
still works (single-arg, equivalent non-sync loop).
Opus adversarial review = SHIP (all 7 findings PASS: determinism [_cp_host_evict_key
purely replicated, pin_expiry dormant under CP], teardown byte-equivalence + no
double-free, no infinite recursion + clean abort-on-partial, snapshot safety [no
ancestor/descendant among host leaves], bounded termination, clean #5 strip,
flag-off path untouched). Tests: new test_eight_ranks_evict_to_fit_stays_identical
(fill->miss->release-in-order->retry -> identical placement, exercises coalescing)
+ 88/88 pool suite (syh-dev-new) + import smoke.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2b.1b adopt-then-strip left the entire option-A commit-quorum apparatus
orphaned: B1's writing_check ReduceOp.MIN frontier (mark_object_committed) is
the all-ranks-done consensus, so the per-(rank,layer,payload) gather quorum it
replaced is information-redundant (design sec 2.4 Corollary). This GC removes
the apparatus and collapses the CpSharedL2PageAllocator commit model to exactly
{_ranges_by_object, _committed_objects, _free_by_payload} -- precisely what
placement_digest() hashes -- so the cross-rank digest assert now covers the
whole model and the dead quorum is structurally unrepresentable.
Removed (all proven dead under B1, confirmed by a full tree caller-sweep +
opus adversarial review = SHIP):
- allocator: commit_layer, _has_full_commit, _expected_layers_for_object_payload,
adopt_reserved_range, set_object_required_payloads; fields _commits_by_object,
_expected_ranks, _adopted_ranges, _required_payloads_by_object,
_expected_layers_by_object, _required_payloads, _expected_layers; ctor args
expected_ranks/expected_layers/required_payloads; module fns
broadcast_cp_shared_l2_decision, gather_cp_shared_l2_{preflight,commits,commit}.
reserve/split_committed_object/_drop_object simplified to ranges+committed-bit.
- cache_controller: the 3 dead _commit_cp_shared_l2_* fns, 3 imports, all 5
option-A ctor params (cp_shared_l2_{cpu_group,source_rank,broadcast_fn,
preflight_fn,commit_fn}), the per-object commit-contract builder/setter/call.
- hiradix: allocator construction updated (3 args dropped); live _split_node
caller and _get_cp_shared_l2_rank_and_group@167 intact.
- tests: split tests migrated to mark_object_committed; 4 dead-symbol tests removed.
Lost __init__ payload-has-slab validation is covered by reserve()'s own
ValueError (fail-loud at first write). Default (flag-off) path unchanged: the
allocator is never constructed when enable_cp_shared_physical_l2_hicache=False.
Net -580 LOC. Validated: 87/87 pool suite (syh-dev-new, torch) + 31/31 core
classes locally + edited-module import smoke. Out of scope (flagged separately):
the CpSharedL2NodeMetadata.{required_payloads,committed_payload_layers} vestige.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The B1 write-through: every CP rank runs the IDENTICAL deterministic CpSharedL2PageAllocator.reserve over the replicated event stream -> identical placement with NO broadcast/gather; commit rides the writing_check ReduceOp.MIN frontier (mark_object_committed) instead of a per-(rank,layer,payload) gather quorum.
cache_controller.py: _reserve_write_cp_shared_l2 stripped to every-rank reserve (dropped preflight#2/rank0-gate/broadcast#3/adopt/missing-payloads; per-object contract kept MF3; idempotent abort-on-partial SF3); _submit_write_cp_layer_states drops the 3 gather-commit calls (per-layer D2H + all-payload done-gate MF1 + per-node ack kept); submit_write_cp_all_layer drops both fallback gathers (zero-owned keeps its ack MF2).
hiradix_cache.py: 3-way merge of l2_pooling shared-L2 init (clean, 0-conflict; Phase-1 clock + E1 + reserve-budget preserved) builds the allocator/slab when the flag is on and passes it to HiCacheController WITHOUT collective fns. The #5 deadlock removed: shared_l2_capacity reserve-miss now SKIPS the backup (reactive, rank-uniform) instead of _evict_host_for_physical_slots(synchronize_across_ranks=True) -- the deterministic shared-pool evict is 2b.2. mark_object_committed wired at _commit_pending_backup (the MIN commit). New _cp_assert_placement_replicated (SGLANG_CP_HICACHE_PLACEMENT_ASSERT) in check_hicache_events: MIN/MAX-reduce placement_digest across the CP group, fail-loud on divergence (Theorem 1 runtime gate; rank-uniform entry, cannot deadlock).
environ.py: SGLANG_CP_HICACHE_PLACEMENT_ASSERT (EnvBool, default off).
Tests: 8-rank reserve-determinism (identical placement across 8 ranks + divergence-detectable) + full pool suite 91/91 + import smoke on g0033 syh-dev-new. TWO opus adversarial reviews both SHIP (cache_controller strip + hiradix wiring).
NOTES: default-off (--enable-cp-shared-physical-l2-hicache); validated for write_through policy (prod default). Capacity-miss currently SKIPS (B1 shared-pool evict = 2b.2) so the flag must stay off in real runs until 2b.2. Dead gather island (3 fns + 3 imports, opus-confirmed inert) pending GC follow-up. Design+proof: docs_internal/cp_hicache_2b_pooled_l2_b1_design.md sec 2.4/9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CpSharedL2PageAllocator.mark_object_committed(object_key) -- the B1 commit path that marks an object committed directly, bypassing the per-(rank,layer,payload) commit_layer/_has_full_commit quorum. Under B1 the all-ranks-done consensus is the writing_check ReduceOp.MIN frontier (the same barrier that releases the node write lock), so every rank calls this with the same object_key at the MIN commit point and the committed set transitions rank-uniformly (feeds placement_digest).
Resolves opus-review MF4: split_committed_object stays coherent for a B1-committed object (no per-layer commit facts) -- it adds children to _committed_objects and the empty _commits_by_object is harmless because B1 never calls _has_full_commit.
7 unit tests (pure-Python): commit-without-quorum, idempotent, unknown-raises, stat bump, digest reflects committed set + lockstep equality, split-of-marked-object coherent, release-after-mark frees pages. Full suite 88/88 green on g0033.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CpSharedL2PageAllocator.placement_digest() -- a deterministic, order-independent SHA-256 of the full replicated allocator state (free list + per-object ranges + committed set). This is the B1 proof-obligation gate from the 2b design (sec 2.4, Theorem 1): the write-through (2b.1) cross-rank assert hashes this each tick and EQ-checks it across CP ranks, turning placement-determinism into a fail-loud runtime invariant -- any reserve/release fired on a subset of ranks or with a per-rank arg changes the digest.
6 unit tests (pure-Python, no torch): determinism, changes-on-reserve, identical-op-sequence => identical-digest, divergent-op => different-digest, reserve+release restores digest (free-list coalescing), placement-order sensitivity. Full suite 81/81 green on g0033 syh-dev-new.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopt the ownerless shared-physical-L2 mechanics from origin/l2_pooling onto the Phase-1 branch, behind --enable-cp-shared-physical-l2-hicache (default off -> behavior-neutral). Foundation for the B1 collective-free allocator (2b.1/2b.2) and the L3 durable floor (Phase 3); no coordination yet.
- cp_shared_l2_pool.py (new): CpSharedL2PageAllocator (deterministic first-fit, capacity charged once, no x cp_size), hugetlbfs-2M slab primitives, position-indexed object ranges, commit-quorum data model.
- memory_pool_host.py: SharedHostTensor* slab orchestration (3-way merged; fix-output's net change to this file is blank-line-only -> additions-only, no content lost).
- server_args.py: flags + _handle_cp_shared_physical_l2_hicache_validation (3-way merged clean with Phase-1 assert + fix-output EnvField).
- test_cp_shared_l2_pool.py: 75 unit tests (allocator/slab/quorum), green on g0033 syh-dev-new; envs stub provides SGLANG_REQ_WAITING_TIMEOUT (read by Phase-1 validation).
Design + B1 proof: docs_internal/cp_hicache_2b_pooled_l2_b1_design.md (sec 2.4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make CP shared-KV HiCache eviction collective-free and scan-resistant by
removing every per-rank wall-clock input from the eviction/admission decisions
(any such input desyncs the must-be-replicated victim set / batch across the 8
CP ranks and can deadlock the collective-coupled writeback/load-back).
- Replicated logical clock: TreeNode.next_access_time() (a process-global
monotonic counter, mirrors mamba/swa's get_last_access_time) replaces
time.monotonic() as the source of last_access_time at every radix bump site
(radix_cache __init__/match/insert; hiradix CP match/insert/_insert_helper_host;
reset() zeroes it). creation_time/pin_expiry intentionally stay wall-clock.
Because match/insert events are replicated (reqs broadcast from rank 0 over the
replicated tree), last_access_time is now identical on every rank. Unique ints
also give a strict total order (no LRU tie ambiguity). No duration arithmetic
reads last_access_time, so non-CP LRU is unchanged; mamba/swa use their own
TreeNode and are untouched.
- CpReplicatedSLRUStrategy = (is_protected[hit>=2], last_access_time, node.id):
scan-resistant (cold one-shots stay probationary, evicted before reused/
protected prefixes), recency-within-segment ages out cold (not LFU), node.id
is the deterministic total-order tiebreak (heaps never compare TreeNodes).
Overridden for CP so it reaches every get_priority eviction site; the host
write-admission key _cp_host_evict_key is switched from FIFO-by-id to it.
- Co-fixes for the same per-rank-wall-clock class: pin_prefix is forbidden under
CP (its pin_expiry victim-eligibility check is per-rank wall-clock; SLRU
scan-resistance covers the benefit); the affinity head_age_s admission input is
disabled (=0.0, relying on the replicated head_defer_count bound); and a
server_args fail-fast guards CP shared-KV against SGLANG_REQ_WAITING_TIMEOUT>0
(its waiting-queue pruning is per-rank wall-clock).
Tests: test_evict_policy.py adds TestCpReplicatedSLRUStrategy (scan-resistance,
recency, total-order, full ordering) -- 30 passed in the dev-cu13 container;
test_radix_cache_unit.py adds logical-clock determinism/total-order tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A long cold-request workload flooded the prefill log (~177MB / 32K lines in
8min, all 8 CP ranks) with cp_host_reservation_plan_insufficient +
prepare_write_backup_reservation_failed even though the host pool was <42%
used and the writes physically fit.
Root cause: _free_room_deficit folds the proactive 20% free-room target
(hicache_host_free_room_ratio) into the admission deficit once a lane dips
below the 10% trigger. _evict_cp_host_for_write_admission then failed -- and
evicted nothing -- whenever it could not reclaim that full 20% target, which
is impossible while the residual is locked/pending. The lane never drained,
so every subsequent backup re-issued the same impossible demand and re-logged
on every tick. The reservation failure is itself a graceful skip, so there is
no crash; the symptom is the log storm.
Gate write-backup admission on the HARD deficit -- max(0, required-available)
over the target and draft lanes (draft_available is 2**62 with no draft pool,
a no-op) -- and evict toward the watermark best-effort, admitting whenever the
write itself fits. This drains the stuck lane and backs the request up instead
of skipping+flooding. The proactive watermark stays as the eviction target
(deficit_by_owner unchanged); it is no longer a hard admission gate. The
decision is computed from rank-replicated state on the collective-free reserve
path, so every CP rank makes the same admit/skip choice (opus-reviewed
rank-safe).
Also rate-limit the four host-reservation fallback warnings (once per 10s per
fallback_name, with a suppressed count) so any genuine exhaustion degrades
quietly instead of producing a log storm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CP shared-KV batch planning needs the immediately previous prepared batch to remain visible as a virtual prefix, but launching another batch before processing that result can leave multiple prepared plans racing against radix insertion. The event loop now processes the previous result after planning the current batch and before launching it, preserving one-batch lookback without accumulating deeper overlap state.
Constraint: CP HiCache prepared batch views are inserted during process_batch_result, not at forward launch.
Rejected: Process previous result before planning current batch | loses visibility of the previous pending prepared plan needed by current planning.
Confidence: medium
Scope-risk: moderate
Directive: Do not increase non-PP overlap depth for CP shared-KV without adding pending-radix reservation semantics.
Tested: Remote cjy-glm5-new pytest test/registered/unit/disaggregation/test_overlap_disagg_prefill_event_loop.py test/registered/unit/disaggregation/test_overlap_disagg_decode_event_loop.py
Not-tested: Full E2E latency impact of reduced overlap depth.
(cherry picked from commit 7df8723eee8203e9e334b229178e3f8bac61396a)
Under overlap scheduling a chunked request's final-chunk write-backup prepare
read a stale `is_chunked` (>0): that per-tick counter is decremented in
process_batch_result, which the overlap loop runs AFTER the run_batch prepare
hook. So prepare floored `backup_end` to a page boundary (the intermediate-chunk
rule) and dropped the now-complete final tail page, while the final non-chunked
insert builds the radix node at full unaligned length. The exact-equality attach
predicate (prepared.logical_len == len(value)) then failed by
(num_tokens-1) mod page_size, dropped the per-layer overlap backup, and forced
the serial all-layer catch-up (~89% of large chunked requests in prod).
Decouple the floor decision from the stale counter: the scheduler marks
`req.cp_backup_is_intermediate_chunk = (req is self.chunked_req)` in
`_prepare_hicache_write_backups_before_forward` (the live chunked_req identity
is the authoritative "will be chunked further" signal, set this tick before
run_batch), and the prepare candidate builder floors on that flag instead of
`is_chunked`. Intermediate chunks still floor; only the misclassified final
chunk now backs up its full tail and attaches the overlap backup.
Tests: update the chunked prepare test to the new flag; add a regression test
that a final chunk with stale is_chunked reserves the full tail; add an offline
repro that drives the real prepare/insert/probe and sweeps unaligned tails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EAGLE verify's traverse_tree.dfs recurses with dfs(retrieve_next_token[curr], ...),
so curr (the fill_next_token_bitmask/accept_token index) degrades from a Python int
to a 0-dim torch tensor at recursion depth >= 2. xgrammar's Python signature is
unchanged across 0.1.27 -> 0.2.1, but the binding swapped pybind11 -> apache-tvm-ffi:
the old binding silently coerced the 0-dim tensor to int, the new one rejects it with
"Expected int but got ffi.Tensor", crashing the decode scheduler on grammar-constrained
(tool-call / structured-output) requests with draft depth >= 2.
Wrap the propagated indices in int(), matching upstream (PR #21722):
- accept_token(int(draft_tokens[curr]))
- dfs(int(retrieve_next_token[curr]), ...)
- dfs(int(retrieve_next_sibling[curr]), ...)
Version-agnostic: correct on both 0.1.27 and 0.2.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EnvField.get() re-read os.environ (os.getenv + parse) on every call. The
attention/compose hot path reads many flags per layer / per compose
(nsa_backend ~11/layer, cp_shared_kv_runtime ~12/compose) -- thousands of
os.getenv + Python-method calls per forward for values that are fixed startup
config.
Cache the parsed value on first read; invalidate on every mutation through the
API (set / override / clear). The only env vars mutated at runtime to pass
information are NOT EnvFields (SGLANG_TMP_NCCL_COMM_VALUE is read by C getenv;
SGLANG_RUN_ID / SGLANG_DP_RANK via raw os.environ), so the cache does not affect
them. Convert the two direct os.environ["SGLANG_*"]=... writes in server_args
(SGLANG_VLM_CACHE_SIZE_MB, SGLANG_ENABLE_DETERMINISTIC_INFERENCE) to envs.X.set()
so they invalidate the cache.
Verified: cache + set/override/clear invalidation behavioral test passes;
331 mem_cache/parser/layers unit tests pass (5 pre-existing CUDA-device-assert
failures unrelated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CP shared-KV cache-hit corruption bug is fixed and verified end-to-end, so
drop the per-layer/per-request trace probes added during the investigation.
Each probe was a Python call plus an UNCACHED os.getenv (EnvField.get() re-reads
os.environ on every call) in the hot path even when disabled -- ~1000 per
forward from the 11 per-layer deepseek probes alone -- plus a latent risk that
an accidentally-set SGLANG_NSA_DUMP_DIR would dump tensors in-forward and tank
throughput.
Delete cp_hicache_trace.py and every reference to it:
- deepseek_v2: 11 per-layer fwd_hash probes + the final dump-flush block
- nsa_backend: the per-compose NSA tensor-dump block
- cp_shared_kv_runtime: 5 imports + the if _cptrace_enabled(...) compose blocks
- cache_controller / hiradix_cache / allocator / memory_pool_host: the
cptrace/khash/knz/rng round-trip + lifecycle hashes
- environ: the SGLANG_CP_HICACHE_KV_TRACE and SGLANG_NSA_DUMP_DIR flags
Kept: SGLANG_DEBUG_CP_SHARED_KV / SGLANG_CP_TRANSFER_LOG transfer logging and the
x-request-id passthrough (independent of the trace module). Verified: imports
clean, 235 mem_cache + reasoning unit tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serving_chat:
- Tolerate a malformed historical tool_call `arguments` string (a valid JSON
document followed by trailing content) instead of 400-ing the whole
multi-turn request: salvage the leading JSON document via raw_decode, else
keep the raw string. (A 112-message tool-history request was rejected with
orjson "unexpected content after document".)
- Catch TypeError (not only jinja2.TemplateError) from the chat-template
render so a `tojson` filter on a Jinja Undefined becomes a clean 400 instead
of a 500 (upstream #20700 / 5e9bd21979).
reasoning_parser:
- Strip only LEADING think-start marker tokens; a global replace would delete a
`<think>` token that legitimately appears inside reasoning content. Preserve
model-generated whitespace in reasoning/normal text (drop .strip()/.rstrip())
(upstream #24251 / dac78768f0).
- Add Glm45 detector tests: leading-only strip, token-inside-content preserved,
repeated leading markers, streaming trailing whitespace.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A bs=1 cache-hit prefill produced deterministically wrong output (e.g. "0.5,0.5,0.5").
Root cause: regression a24111a5f4 changed the bs=1 cache-hit call sites to pass
prefix_slot_spans=[] (get_or_build_batch_slot_spans want_prefix=False) together with
prefix_pages>0. In materialize_prefix_and_reuse_current_kv_page_slots and its index twin,
the guard `if prefix_slot_spans is not None:` let the empty list shadow the prefix_pages
fallback -> prefix_spans=[] -> the cached prefix dense slots were never materialized
(both IPC and local paths gate on `if prefix_spans:`) -> attention read zero KV over the
entire reused prefix. The indexer compose dropped its prefix the same way. bs=1 only;
bs>1 (non-empty per-request spans) and fresh prefill (no prefix) were unaffected.
Fix: make the canonical per-request slot-span list the sole description of the
prefix/current regions and fail loud on a missing list. Remove all four
span-reconstruction fallbacks: prefix_slot_span (singular, dead), prefix_pages->span
(the bug), current_slot_spans=None->span (dead), and the prefetcher's copy. Both twins
now require prefix_slot_spans + current_slot_spans (raise on None; [] = genuine
no-prefix). All call sites build canonical spans via want_prefix=True (cached per-forward,
so a24111a5f4's per-layer-CPU goal is preserved by the cache, not by skipping the build);
the MLA current-only path also uses want_prefix=True to keep want_prefix uniform and the
span cache thrash-free.
Tests: migrate unit tests off the removed prefix_pages param (drop it where spans were
already passed; supply canonical spans otherwise); fix two source-string/captured-kwarg
assertions; add test_materialize_prefix_requires_explicit_prefix_slot_spans (None->raise,
explicit spans -> non-zero prefix). test_cp_shared_kv_runtime.py: 158 passed on the
g0033 container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SGLANG_DEBUG_CP_SHARED_KV also disables tai IPC materialize -> NSA index fail-fast at
warmup (unusable in this config). Add a logging-only flag that emits the sender/worker
transfer-partition dumps (main-KV pages/positions vs NSA-state pages/positions) without
that side effect, and lift the 64-log cap, so we can diff the state-vs-main-KV partition
for a cache-hit vs cache-miss.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rid 'dumpfh-' -> dump ONLY the model's final hidden (last token, post-CP-gather = the
first-token source), allowed on FRESH prefills too -> the cache-bust-vs-cache-hit
discriminator (prefill-side vs decode-side). rid 'dump-' keeps the lean per-layer
trajectory (cache-hit only). Flush moved after the CP gather. dump_diff generalized to
compare any two phases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The full-stage+kv dump (1.7GB/file, synchronous in-forward) hung/crashed a scheduler
child. Trim to the residual-stream trajectory (attn_in/attn_out per layer) + q_all +
final_hidden (~tens of MB/forward) — enough to localize the first divergent layer (Stage 1).
Drop dense-space kv_cache/topk (not comparable anyway) and the MoE substages. Dumps go to
local NVMe (/ssd) to avoid beegfs stall.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dump_tensors now ACCUMULATES per-(layer,stage) into a forward buffer (every fwd_hash
stage at every layer + nsa q_all/attn_out/topk all layers, kv_cache at a layer spread);
dump_flush writes one file per (rid,rank) at model end with the full trajectory + final
hidden + positions + seq/prefix metadata. Gated to cache-hit extends only (skip fresh
full prefills) so --repeat 2 dumps just the L1-hit + L2-reload (small). dump_diff joins
A(rep1) vs C by idx, relerr per token-order stage -> first divergent layer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- http_server: chat + generate endpoints read x-request-id header into rid (the
Rust PD router drops the client body rid but forwards the header), so the
client id reaches forward_batch.rids for exact cross-send join.
- cp_hicache_trace.dump_tensors + SGLANG_NSA_DUMP_DIR: torch.save q/composed-KV/
selection/attn_out at layer 0 for rids starting 'dump-' (extend forwards), to
diff L1-hit vs L2-reload by relative error (beats fp-nondeterminism that
defeats binary hashes). Wired into nsa_backend flashmla_sparse path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a per-request content fingerprint (hash of extend input-ids + seq_len) to
fwd_hash so the SAME request forwarded from an L1-hit (known-good) and an
L2-reload (suspect) can be JOINED across the log without rid (the Rust PD
gateway strips the client rid). Gated to bs<=1 forwards (the join is only
meaningful single-request, and this skips the c=24 flood forwards so the
level-3 log stays small). The analyzer joins by ck and reports the first
DETERMINISTIC stage (topk/attn) that diverges = corruption localized.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The decisive level-1 instrumentation for the cross-rank logical-loc
divergence hypothesis. Prior traces all keyed PER-RANK (host pools aren't
comparable across ranks) -- which structurally hid the one invariant that
matters: the visible LOGICAL-loc sequence for a reloaded node must be
bit-identical on every CP rank, or the cross-rank gather reads token j from
its owner rank at a slot that rank filled with a different token.
- visible_locs_hash (cache_controller.load_cp): per-node, per-rank khash of
the visible logical locs, emitted for EVERY node incl. zero-owned so the
analyzer can diff across ranks by node_id. Divergence = root cause.
- victim_set (hiradix _evict_cp_load_back_owner_lanes): per-rank device
load-back eviction victim ids. Device eviction is keyed on wall-clock
last_access_time (per-rank, NOT replicated -- unlike host eviction's
deterministic (priority, node.id)); divergent victims is the trigger.
- release_draw (allocator.alloc_pages_with_owners): fires when a lane's free
bucket is exhausted and the deferred-free release bucket is tapped -- the
use-after-free/aliasing secondary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two metadata agents verified that all cache-hit-extend metadata (positions,
page_table, cu_seqlens, topk offset, MoE row-selection) is correct and
prefix-aware. The one reload-specific, unguarded thing is the per-page OWNER
replay: load_cp builds selected_logical_locs = node_device_indices[owned_positions]
and explicitly SKIPS re-validating that those pages are owned by this rank (perf;
the allocator's owner assert is gated behind debug_mode). A per-page owner
permutation (correct bytes, wrong logical page) passes the aggregate host-keyed
value-hash but makes attention read the right bytes for the wrong positions ->
reload-only repeated-token garbage.
Add owner_check (level 1): at each reloaded node, owner_for_logical_pages(
selected_logical_locs // page_size) must all == cp_rank; log bad_owner count +
valid/padded/tail_pad. bad_owner>0 = the per-page owner/padding permutation =
root cause, which the aggregate byte-hash cannot detect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both cached KV components round-trip byte-perfect, so the bug is downstream in the
cross-rank COMPOSE/GATHER on a reload forward. Add level-3 hashes targeting the
active suspect = materialize_prefix_and_reuse_current_kv_page_slots (reloaded
prefix gathered via modulo-owner IPC descriptors + fresh current via page_inverse
staging; the abutting prefix|current boundary is the suspect):
- gather_out: the live composed dense KV the attention actually reads (mixed_locs
gather), with prefix_pages/current_pages; nz==0 = zero/uninitialized composed KV
(the observed 0|0|0), self-contained.
- span_owner: per prefix/current span -> owner ranks (modulo) + physical pages +
logical pages + per-span hash/nz, to verify the prefix span maps to modulo
owners and catch a boundary conflation/off-by-one.
- ipc_desc: owner/src/logical-page ranges in build_cp_shared_kv_ipc_page_descriptors
(the modulo-owner prefix gather chokepoint).
- MoE stages (forward_deepep, via fwd_hash, valid-row local tensors only, never the
a2a-permuted intermediate): moe_postsel, router_logits, topk_ids/topk_w,
experts_out, moe_out_compact, moe_out_restored -> brackets select/router/topk/
dispatch-combine/shared-add/row-restore.
(The symm ComposePlan is dormant/unwired, so excluded.) All level 3, eager-extend
only, try/except-guarded, helpers khash/knz/rng. Analyzer flags zero composed KV +
zero spans + MoE-stage zeros.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Main KV round-trips byte-perfect yet output is garbage, so add the two
unverified components:
1. NSA INDEXER-K round-trip (level 2): hash the device index_k_with_scale_buffer
at backup (_backup_indexer_from_device_per_layer) and reload (NSA
load_to_device_per_layer, after _load_indexer), keyed by host-slot fingerprint
+ layer, with khash+nz. The indexer selects which tokens attention attends
(top-k); if it corrupts on reload -> wrong selection -> garbage even with
correct main KV.
2. FORWARD-side per-layer hashes (level 3, eager extend path only, cuda-graph
guarded): attn-input, attn-output (pre-residual), topk_indices (the indexer's
selection output -- direct consumer of the indexer-K), and MoE-input, in the
DeepseekV2/GlmMoeDsa decoder layer forward. Localizes where a reload forward
diverges: topk diverges => indexer-K cache; attn-out diverges (topk ok) =>
main KV/page mapping; moe-in diverges (attn-out ok) => residual/MoE.
Analyzer compares indexer reload vs backup (host_fp keyed) + flags zero/degenerate
hidden states per stage. Level 3 (SGLANG_CP_HICACHE_KV_TRACE=3) captures everything.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first KV-value pass couldn't compare (reload hashes keyed by the merged
load op's node id, backup hashes per node -> no join). Fix: emit the host-slot
range fingerprint at both backup and reload (host slots are the stable identity
across node id / splits / merge), and the analyzer matches on
(rank, layer, host_fp). Also add a nonzero-byte count (nz): real KV is never
all-zero, so an nz==0 slice is uninitialized/zero KV -- a direct, self-contained
corruption flag that needs no cross-stage matching. This targets the observed
'0|0|0...' garbage on large-cached reloads (store never filled the backed-up
pages, or the round-trip lost them). Gated at SGLANG_CP_HICACHE_KV_TRACE=2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The index trace proved per-node addressing is self-consistent, so the
corruption must be in the KV VALUES, not the bookkeeping. Add a full-tensor
position-weighted fingerprint (khash) of the actual device KV at two points,
both keyed by (node_id, layer_id) in owned-position order so they compare
directly:
- backup_kv_hash: device KV at backup, taken on the DEFAULT stream (outside the
write_stream block) so it is the correct POST-STORE value, not the racy
write_stream copy's view.
- reload_kv_hash: device KV after the H2D load (on load_stream, in-order after
the copy).
If a reload hash matches no backup hash for that node+layer, the round-trip
delivered wrong KV -- catching BOTH a per-layer store-vs-copy ordering race
(backup-correct != reloaded-stale) AND transfer corruption, in one run. Since
the compose/attention is identical fresh-vs-reload and stateless-per-forward,
all-ranks byte-correct shards => correct output, so this per-rank value check is
complete for the round-trip. Gated at SGLANG_CP_HICACHE_KV_TRACE=2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Heavily-forked CP HiCache produces KV corruption only after an L1->L2->L1
round-trip (eviction + reload), scaling with cached volume. Static reading and
upstream-diff are exhausted, so trace the data flow and check four invariants
directly, gated by SGLANG_CP_HICACHE_KV_TRACE (0=off, 1=structural lifecycle,
2=+compose/free/ack), off by default and cheap when off.
Trace points (keyed by node_id; rid_map ties client rid->node_id):
- backup_reserve / backup_d2h: host<->phys slots written (H2/H4)
- split: prefix+suffix partition vs parent (H1)
- evict: write_pending at device free (H4)
- reload_node: host slots read on reload (H2)
- reload_assign: reloaded device locs -> forward
- dev_free / write_ack: device free vs backup-complete ordering (H4)
- compose / remap: per-forward row-id cache reuse + unmapped count (H3)
New mem_cache/cp_hicache_trace.py (cptrace/rng helper; values rendered
space-free for offline parsing). The companion analyzer joins these by
node_id; the decisive, request-independent check is H2: any node whose reload
host-slot fingerprint differs from its backup fingerprint is the corruption
source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build_flattened_request_row_ids ran once per layer in the per-layer
prefill attention loop and rebuilt the indexer-seq-len-derived row ids
on the GPU via repeat_interleave with a *device* repeats tensor, which
forces a device sync (it reads sum(repeats) to size the output) and
serializes the launch thread every layer.
The row ids depend only on indexer_seq_lens_cpu, which is identical
across every layer of a forward pass. Add
get_cp_shared_kv_flattened_request_row_ids, which builds them once and
caches on the ForwardBatch (keyed by expected flattened length + device
so a different batch shape or the draft-vs-target pass never reuses a
stale tensor), and build on CPU then move once to drop the sync.
In a TP-5 warm cachebench trace this collapsed 1817 wrapper calls to 46
actual builds (97.5% fewer) and dropped ipc_materialize visibly.
GSM8K 200q x2 = 0.955 / 0.950, 0 invalid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EAGLEWorkerV2's target prefill always captured the global FULL hidden,
so the bs>1 NSA CP shared-KV draft extend received a global hidden whose
row count (real total tokens) matched neither the per-rank CP-local count
nor the MLP-sync CP-aligned padded count, tripping
[CP_SHARED_KV_FAIL_FAST][draft_batch_gt1_spec_hidden_shape_mismatch]
under SGLANG_ENABLE_SPEC_V2=1.
Mirror the legacy EAGLEWorker contract: add _can_use_cp_draft_shared_kv
and, when CP draft shared-KV applies, capture the CP-local hidden side
channel (CaptureHiddenMode.NULL + capture_draft_hidden_states) instead of
FULL. The v2 consumer already prefers draft_hidden_states when present
(commit 5e22279670 added the consumer but not this producer side).
Fixes the shape at the source rather than loosening the NextN fail-fast.
CP-off keeps FULL; non-CP / bs=1 unaffected; legacy v1 path untouched.
Validated on g0033 PD (prefill + 2-node decode): bs>1 up to 17 incl. the
former-crashing bs=6, 0 non-400 errors, GSM8K 0.965 / 0 invalid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deferred non-chunked inserts can happen while a CP HiCache per-layer backup is still in flight. Rolling back the unattached prepared backup at unfinished-request time drops the only reservation that final insertion can attach, so cache_finished_req has to start a post-forward backup and catch up all layers synchronously.\n\nKeep the prepared backup for non-chunked deferred inserts and continue rolling it back for chunked middle inserts, where the suffix may be extended by later chunks and the old backup would cover the wrong range. Document the failure mode so future changes do not rediscover the same fallback path.\n\nConstraint: CP HiCache radix split cannot mutate in-flight backup nodes.\nConstraint: Chunked prefill middle inserts still need rollback because their backup range is not final.\nRejected: Always rollback unattached backups | causes post-forward catch_up_all_layers for non-chunked deferred inserts.\nRejected: Always preserve unattached backups | can attach stale backup ranges for chunked middle inserts.\nConfidence: high\nScope-risk: narrow\nDirective: Do not clear req.cp_hicache_prepared_backup on non-chunked deferred insert without proving final insert no longer needs it.\nTested: remote cjy-glm5-new py_compile radix_cache.py\nTested: remote cjy-glm5-new pytest test_cp_hicache_metadata.py::{nonchunked preserve,chunked rollback} => 2 passed\nNot-tested: live ETE replay after restarting prefill with this exact commit
