A single cudaHostRegister over a >~1 TB host buffer fails with
cudaErrorMemoryAllocation on B300 (hard per-call ceiling: 512 GiB OK, 1024 GiB
FAIL), crashing the hicache_size=1600 (~1.5 TB CP shared-L2 slab) prefill at
startup. The registration cannot simply be chunked: a memcpy (cudaMemcpyBatchAsync,
the CP-L2 H2D/D2H transfer) fails with cudaErrorInvalidValue when its host range
straddles a registration boundary (verified empirically on b300-049).
Fix: physically split the host cache into multiple page-aligned slabs, each <= a
safe single-registration size (default 480 GiB, env SGLANG_CP_HICACHE_MAX_SLAB_GB),
reusing the existing SharedHostTensorGroupAllocator + per-slab transfer splitting
(_host_transfer_segments). Each slab is one whole registration and no transfer
crosses a boundary; small configs (hicache_size<=400) stay single-slab unchanged.
- memory_pool_host.py: add cp_hicache_max_single_register_bytes() + the fail-loud
_check_single_cuda_host_register_size guard; revert the (transfer-unsafe)
registration chunking back to one cudaHostRegister per buffer/slab.
- hiradix_cache.py: _cp_shared_l2_slab_pages_by_payload auto-caps each payload's
slab <= the ceiling so large caches auto-split.
- cp_l3_slab_accessor.py: CpSharedL2SlabAccessor is now slab-count-aware
(CpL3SlabSpan + per-slab dispatch via global_base_page; per-slab layer stride);
_cp_l3_slab_spans rewires _maybe_init_cp_l3 off the single-slab assumption. L3
disk slabs / slot pool / LMDB index / GC are content-addressed and unchanged.
Tests: multi-slab accessor incl. a non-circular torch.frombuffer-layout check; a
real allocate_group + _cp_l3_slab_spans roundtrip; slab-cap auto-split; L3 store
cross-slab spill/reload. Reviewed by 3 adversarial agents, no correctness bugs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The host-pool size check used psutil.virtual_memory().available -- regular RAM -- unconditionally. But
hugepages are carved OUT of regular RAM, so reserving e.g. 1 TB of 2M hugepages drops psutil.available
by ~1 TB, and the check then spuriously fails ("Not enough host memory ... only have 897 GB free") even
though the slab is allocated from the hugepages, not regular RAM. It was inspecting the wrong pool.
Two fixes:
- memory_pool_host.py: gate the psutil check on the DEFAULT (regular-malloc) allocator. A custom
host_tensor_allocator (the CP shared-L2 slab over hugetlbfs) owns its own capacity check, so skip the
regular-RAM check for it.
- cp_shared_l2_pool.py: add cp_shared_l2_hugetlbfs_free_bytes() + check_cp_shared_l2_hugetlbfs_capacity()
(statvfs the hugetlbfs mount: f_frsize = hugepage size, f_bavail = free hugepages) and call it in
create_cp_shared_host_slab before ftruncate/mmap. Now insufficient hugepages fail LOUD with a clean,
actionable message ("insufficient hugepages in /dev/hugepages: need X GB (N x 2M pages) but only Y GB
free; reserve more or reduce --hicache-size") instead of a cryptic mmap ENOMEM. Per-slab + sequential,
so each payload's slab sees the free remaining after the prior ones.
Unit-tested (capacity check: tiny fits, over-capacity fails loud). Pre-existing 2b L2-pooling code (not L3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The collector was created only in _maybe_init_cp_l3 (enable_cp_l3 path), so a pure-L2-pooling
deployment (no L3) got no L2 metrics -- exactly the config under perf investigation. Move creation
into _cp_maybe_collect_metrics: lazily create on first emit, gated on enable_metrics + the shared-L2
allocator existing (L2 pooling on). collect() already handles l3_stats=None, so it reports L2
occupancy/commit/evict with or without L3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under CP, enable_storage=False, so StorageMetricsCollector (prefetch/backup tokens + bandwidth) never
emits, and L2 pool usage was debug-log-only (_cp_maybe_log_lane_stats) while L3 was invisible. Add a
CpHiCacheMetricsCollector covering both, gated on enable_metrics alone, per-rank via the tp_rank label.
- CpL3Store.stats(): per-payload slot occupancy (used,total) + cumulative counters (spill_pages,
reload_pages, gc_reclaimed, write_errors -- incremented on the bg threads) + instantaneous inflight
(spill/reload) + the four bg queue depths. Counters are monotonic (NOT reset on clear()).
- CpHiCacheMetricsCollector (metrics_collector.py): gauges for instantaneous state
(sglang:cp_l2_used/total_pages, cp_l3_used/total_slots, cp_l3_inflight_*, cp_l3_queue_depth) +
counters for cumulative work (cp_l2_objects_committed/evicted_total, cp_l3_spill/reload_pages_total,
cp_l3_gc_reclaimed_total, cp_l3_write_errors_total). Counters fed via a monotonic .inc(delta) off the
store's cumulative totals.
- hiradix: create the collector in _maybe_init_cp_l3 (once, per-rank labels from cache_controller);
push periodically (~1s, per-rank wall-clock -- metrics need no rank-uniformity) from
check_hicache_events via _cp_maybe_collect_metrics, reading allocator.occupancy_by_payload()/stats()
+ store.stats(). Off the data path.
store.stats() unit-tested (occupancy + spill/reload counters; 10 store tests green). The collector +
wiring are correct-by-pattern (mirror StorageMetricsCollector) and validate on the live server (3.4;
prometheus_client not in the unit env). Directly useful for 3.4: watch L2/L3 fill + GC + reload hits
under cachebench.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces §9.3's object-level rank-uniform eviction with per-rank, page-level LRU GC running ON THE
WRITE THREAD (the sole slot-pool owner -> no lock, no second owner). Genuinely communication-free:
each (hash,payload) is written/read/deleted by exactly its one owner (owner = i%cp_size, and the
prefix-chained content hash makes one-hash = one-position = one-owner), so the shared LMDB's own
consistency substitutes for collectives within L3 (design §9.5/§9.6). GC touches only L3, adds zero
collectives; the spill/reload MINs that remain are all the L2 bridge.
cp_l3_store.py:
- Per-payload lazy-deletion min-heap `_gc_heap` of (last_access, seq, hash) + authoritative
`_gc_current{hash->(last_access,slot)}`. Add on write, touch via `_touch_q`, reclaim coldest.
- The write loop, each iteration: drain touches -> _gc_collect (reclaim coldest of any pool over the
0.90 start watermark down to 0.85, bounded budget) BEFORE the next write, so spill never hits a full
pool (no evict burst on the critical path). Continuous + proactive, symmetric with the spill.
- Reclaim deletes the index entry (durable) BEFORE freeing the slot, so a racing reload reads None or a
CRC-mismatched reused slot -> miss -> recompute (fail-soft); no lock / in-flight-exclusion needed.
- submit_touch (scheduler->write thread); clear() resets the GC structures.
hiradix_cache.py: wire the reload `touch` (the 3.2 gap -- hit_count/last_access never updated): at
_cp_l3_admit_reload, bump the L3 last_access of THIS rank's owned reloaded pages (ALL pages, not just
the tail, so a prefix ages uniformly). _cp_l3_insert_reloaded_node now returns the node (its
last_access seeds the touch).
Sharing handled optimally for free (page = one entry, last_access = max over touchers; prefix pages
age together; shared-hot survives) -- no object-record table, no refcount, no content-keying. 56 L3
unit tests green (new: GC reclaims coldest-to-watermark + no slot leak; touch keeps a page warm).
Cold-rebuild (scan slab headers + LMDB) is designed (§9.6) but deferred to 3.4 (crash-recovery).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a radix miss, reload the evicted-but-L3-durable suffix from disk into L2 + insert a fresh
radix node, holding the triggering request until then (mirrors the storage-prefetch hold, which
is disabled under CP). The reloaded prefix re-enters the radix as a normal L2 node, so the
existing load_back serves L2->L1 -- no new device path.
Design (docs_internal/cp_hicache_l3_phase3_impl_design.md §9, refined for minimal sync):
- ENTRY (scheduler _prefetch_kvcache, enable_cp_l3 branch): mark a miss-with-suffix as a reload
candidate (rank-uniform: requests are broadcast). cp_l3_reload_lookup computes the suffix's
full-page content hashes -- the SAME SHA chain spill wrote (compute_node_hash_values).
- RESERVE + ADMIT fold into the existing per-tick _drain_l3_control_queues MINs -- ZERO new
collectives: the candidates' per-rank exists_prefix counts ride the qsize MIN vector -> a
rank-uniform agreed count C (this reconciles the async-LMDB read-skew); reload-ack oks ride the
durable ok-AND. Reserve is then collective-free (deterministic evict-to-fit mirroring
_reserve_write_cp_shared_l2_evict_to_fit); the request waits non-blocking in waiting_queue
(check_cp_l3_reload_progress skip) and is admitted + re-matched the SAME tick the reload lands
(check_hicache_events runs before batch formation). CP-aware insert only on a still-clean attach
(else abort+recompute). Capped (cp_l3_reload_max_inflight) + content-key deduped (piggyback).
EAGLE/bigram (opus-studied, PROVABLY correct -- not deferred): the radix key is bigram-converted
over the whole request, and convert_to_bigram_key(fill_ids[M:]) == bigrams(fill_ids)[M:] exactly
(the boundary bigram belongs to the matched prefix), so the bigram-converted suffix hashes
reproduce node.hash_value. Verified vs source + a new slice-identity regression test.
Opus review (FIX-THEN-SHIP, no CRITICAL; the 4 crash surfaces -- MIN-vector shape, read-skew,
attach-anchor, placement-digest -- traced clean) + independently re-verified; fixes folded:
- M1: cp_l3_release_request clears reload state on request abort/timeout/preempt (no leak).
- H1: rank-uniform per-op TTL bound releases a held request if its reload op never acks (the
default SGLANG_REQ_WAITING_TIMEOUT is off); the reservation frees safely at the (late) ack.
- L1: fail-soft anchor guard at insert (re-derive the first suffix hash from lhn's live hash).
- empty-waiters reload aborts (frees L2) instead of inserting an unrequested node.
Imports hoisted to top-level (get_hash_str -- hicache_storage is a leaf module, no cycle;
convert_to_bigram_key already top-level), no inline imports. py_compile clean; 54 L3 unit tests
green. Reload triggering (submit_reload/exists_prefix/free_object) now wired (was write-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first 3.1 (d08ee9f8d) spilled the COLDEST resident objects and held the
protect_host eviction-pin through the entire slow disk write (released only at the
durable ack). Since the coldest objects ARE eviction's victims, a B300 cachebench
wire test wedged: evict-to-fit found everything pinned (evicted=0) ->
host_reservation_failed flood -> caching broke.
Redesign (docs_internal/cp_hicache_l3_phase3_impl_design.md §R3):
- cp_l3_store: split the single spill thread into a GATHER -> WRITE pipeline. Gather
copies each owned page off the pinned slab into a slab-independent anonymous-mmap
staging buffer (fast RAM memcpy) and acks GATHER; a separate write thread does the
O_DIRECT write + fdatasync + durable LMDB put off the staged copy and acks DURABLE.
The eviction-pin releases at the GATHER ack, so the object is evictable right after
the RAM copy; the disk write completes off the staged copy even if L2 reuses the
pages. Slab-full frees its partial slot allocation (no orphan) + acks ok=False.
- hiradix: continuous PROACTIVE spill — enqueue each just-committed object to a FIFO
deque at the replicated commit frontier (_commit_pending_backup) AND at radix splits
(the freshly-committed parent half), drained in commit order (not coldest). The deque
is capped (rank-uniform) so a disk stall can't OOM it. Spilling HOT just-committed
objects (disjoint from eviction's COLD victims) means the cold tail is already
L3-durable when evicted -> eviction just drops it, never contends for the pin.
- 3-element CP-group MIN drain [gather, durable, reload]: gather-MIN -> release_host;
durable-MIN -> l3_durable, gated by an ok-AND (a second MIN over per-op ok bits) so a
write failure on any rank means NO rank marks it durable (rank-uniform durability,
placement_digest stays green).
opus-reviewed (FIX-THEN-SHIP) + independently re-verified; review fixes folded:
split-enqueue (HIGH), deque cap for the disk-stall OOM (MED), fail-soft-on-write-failure
documented (HIGH; deliberately NOT re-enqueued — re-gather would churn under slab-full,
recompute is correctness-safe), rate-limited error logs (MED/LOW). 54 L3 unit tests
green (new: 2-phase gather-before-durable ordering, slab-full fail-soft + no slot leak).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spill is decoupled from eviction (approach D, user-confirmed): a rank-synced background maintainer at
check_hicache_events COPIES the coldest resident committed objects (replicated SLRU order, not-yet-in-L3,
unpinned) to L3 off the critical path. The object stays L2-resident; protect_host pins the LIVE node during
the async bg gather (no freed-range hazard). owned_pages owner = object-local i%cp_size (verified vs
_shared_l2_current_page_owners); per-payload base_page from object_ranges; logical pages only. Bounded +
backpressured (cp_l3_spill_max_inflight). Replicated selection -> lockstep object-granular acks. Spill-ack
(in _drain_l3_control_queues): release_host + node.l3_durable=True (O(1) skip; memoized from exists_prefix
for reloaded nodes). Eviction unchanged (drops L2; L3 copy survives -> reload in 3.2; not-yet-spilled =
recompute, fail-soft). Compiles; default-off. Live validation via flag-on smoke (branch sync).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_maybe_init_cp_l3: build the 3 CpSharedL2SlabAccessor (target_kv always; draft_kv if draft pool;
index_k if NSA) from the LIVE host pool slabs (verified attrs: host.allocator.mapping.mmap,
layer_num/page_num/kv_cache_dim/dtype.itemsize; index_host_tensor_allocator + indexer attrs), then
CpL3Store.from_config + connect (PLP gate). Single-slab only (default --cp-shared-l2-slab-size-gb 0);
fail loud on the multi-slab group case (future extension, not a silent stub).
_drain_l3_control_queues: per-tick CP-cpu-group MIN over the 2 ack-qsizes + drain MIN of each, idle
early-return on the replicated has_inflight (R1 CRIT-3/4) — wired in check_hicache_events (gated on
enable_cp_l3, NOT the enable_storage drain). reset() -> cp_l3_store.clear(); shutdown() -> close().
Default-off + behavior-neutral. Compiles. 3.0 CODE complete; flag-on smoke gated on cluster branch sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
server_args: --enable-cp-l3 + --cp-l3-config fields/args. Validation (R1 HIGH-2): do NOT re-gate
the 6 CP storage forbids (they gate on hicache_storage_backend and correctly never fire); instead
assert enable_cp_l3 is mutually exclusive with hicache_storage_backend, requires
enable_cp_shared_physical_l2_hicache, and requires cp_l3_config.
hiradix: self.enable_cp_l3 (CP-only) + cp_l3_store placeholder; compute_node_hash_values now also
fires under enable_cp_l3 (the per-page content hash is L3's key) — clean-boot only (R1 HIGH-3).
Default-off + behavior-neutral; store construction + drain follow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ties config -> per-rank disk slabs + slot pools + shared LMDB index + slab accessors. Background
spill/reload threads (off the scheduler tick, storage-template shape); object-granular acks (one per
object after all owned pages durable; zero-owned ranks ack in lockstep) via ack queues + has_inflight,
so the caller's CP-cpu-group MIN-drain frees the same objects on every rank. Spill = gather->O_DIRECT
write->durable index (data->fdatasync->index-commit ordering) with content-hash dedup; reload =
index lookup->read_into->scatter (verify-on-read). free_object frees owned slots+index (3.3 eviction
consumes); clear() is the flush_cache hook (stop/drain/reset/restart). from_config splits the disk
budget across ranks-on-disk x payloads (equal slot count). PLP gate at connect(). Model B: store is
content-addressed + rank-local I/O; only the LMDB is shared. End-to-end test (spill/exists/evict/reload
byte-exact/dedup/free/clear) 5/5 (venv lmdb). + accessor n_layers/page_num accessors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CpSharedL2SlabAccessor (HIGH-1): memcpy-gather a page's N_layers strided slab slices into a
contiguous buffer (spill) + scatter a contiguous blob back into the strided slices (reload) —
the cheap RAM bridge that lets host stay layer_page_first (inference perf) while disk is
page-contiguous. Generic CpL3SlabLayout(n_layers, page_num, slice_bytes) with verified-from-source
factories: for_mla (layer_num,page_num,page_size,kv_cache_dim,itemsize -> 2.74 MiB/page) and
for_index (n_active_layers,indexer_page_num,indexer_page_stride -> 0.17/0.63 MiB/page). gather_into/
scatter_from take an offset so the spill writes straight after the 64B slot header (zero extra copy).
Pure (mmap+ints), no torch. 8/8 tests. Canonical layer 0..N-1 order shared by spill+reload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Preallocated disk-slab file per (payload_kind, drive); a page = one contiguous blob
(64B header + gathered payload + pad) in a 4K-aligned slot. O_DIRECT single-transfer
write/read (measured best vs vectored), buffered+fdatasync fallback for FS without O_DIRECT.
write_slot/read_slot convenience + write_aligned/read_into zero-copy (the QD>1 spill/reload
path gathers/scatters straight into the aligned slot buffer). verify-on-read: absent/torn/
foreign slot -> None (a MISS to recompute + reclaim), never a crash. posix_fallocate preallocate.
9/9 unit tests (real temp FS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CpL3Config (TOML/JSON): per-machine L3 disks (paths + budgets), backend, shared index
location/size, and the rank->disk mapping. disk_for_rank balances M CP ranks over N disks
(r % N; ranks share a disk gracefully — throughput is drive-count-bound) or honors an
explicit map; fail-loud on length/range mismatch. probe_disk_plp: O_DIRECT write+fdatasync
latency heuristic for the durability gate (non-PLP disk must fail loud when require_plp).
No hardcoded paths. 11/11 unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3.0 (L3 disk durable floor) foundational primitives, pure + unit-tested:
- 64B self-describing blob header (magic/version/payload-kind/content-hash/CRC),
verified on every read (verify_blob) -> torn/bit-rot slots become a MISS, not garbage.
- slot_bytes_for_page: header+payload rounded to the 4K O_DIRECT boundary.
- CpL3SlotPool: O(1) free-list over fixed-size per-payload disk page-slots (the L3
analog of CpSharedL2PageAllocator's free list), fail-loud on double-free/OOB, with
reset() for the flush_cache hook. NOT a ring buffer (eviction frees arbitrary slots).
Rank-local (a rank stores only its owned pages; L3 eviction selection rides the shared
LMDB replicated clock). 12/12 unit tests. Design: docs_internal/cp_hicache_l3_phase3_impl_design.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the interim ack_queue_len==0 band-aid with a structural fix so the perf skip is
SAFE rather than removed.
Root cause: the radix attach/defer/prune/split probes read `_node_host_write_pending`
(membership in ongoing_write_through/pending_host_backups, which are drained ASYNCHRONOUSLY
at per-rank ack completion), and `_cp_subtree_has_unprunable_state` also read the rank-local
`lock_ref`. So per-rank ack-drain timing leaked into the match_prefix truncation ->
cache_protected_len -> logical_len/len(value) -> the attach predicate. On some ranks the
prepared backup attached, on others it was dropped to the synchronous write_backup catch-up
(catch_up_all_layers draft=True), whose ack-append is per-rank. ack_write_queue then diverged
across CP ranks, so writing_check's ack_queue_len==0 early-return diverged: a strict subset
entered the writing_check_min all_reduce while the rest advanced to recv_requests' broadcast
on the same tp_cpu_group (8 ranks; dp_size==1 resets enable_dp_attention to False) -> NCCL
deadlock (prod hang: node_id=50713, all 8 GPUs 0% util).
Fix: introduce TreeNode.cp_backup_pending, a rank-replicated marker set at the rank-replicated
prepared-backup attach (_attach_prepared_cp_backup) and the guarded write_backup registration,
cleared only at the MIN-committed commit (_commit_pending_backup) / rollback. Rewire the CP
radix probes (_cp_node_split_still_pending, _split_node guard, _cp_subtree_has_unprunable_state,
_inc_hit_count) to read this marker instead of the drain-timed dicts, and DROP the lock_ref read
in the prune probe (its only cross-rank skew is the backup lock, now covered by the marker).
_node_host_write_pending stays the drain-timed accessor used only by writing_check / eviction /
the visibility check (_node_host_write_ready). With the radix-path reads replicated by
construction, len(value)==logical_len on every rank -> the prepared backup always attaches (or
rolls back) symmetrically -> the asymmetric catch-up never fires -> ack_write_queue is symmetric
-> the existing ack_queue_len==0 skip is all-or-none (no per-tick no-op MIN, best perf).
Correctness: the marker is cleared only at/after _commit_pending_backup, which runs only under
the MIN frontier, so the MIN remains the SOLE host-visibility gate (no KV-corruption class
reopened); the marker is strictly more conservative than the old dicts (stays True until commit).
Fail-fasts: double-attach raises; _split_node propagates the marker. hi_mamba unchanged (gates
on ongoing only, no async ack skip). Single-layer-draft (c3fc3ff752) preserved: the design
touches only the attach decision; the draft notifier already fires post-MoE.
Adds test_cp_hicache_symmetric_attach.py: the probes read the replicated marker not the dicts,
the prune probe ignores lock_ref, and the marker lifecycle (attach->commit->rollback) + the
double-attach guard. All pass on the fix; all fail on the pre-fix code.
Design: docs_internal/cp_hicache_symmetric_attach_design.md. Pending: ETE (GSM8K + no-hang
flood + TTFT) on the dev-cu13 container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 876a98177cd749b5a63623ba2ae8b868289b8e59)
With DeepEP the sparse-MoE layer keeps tokens SCATTERED (DeepEP dispatch/combine
routes tokens to expert-owning ranks), so the NSA layer communicator never
materializes the FULL hidden states. Decoupling CP onto the standard/allgather
a2a (flashinfer_trtllm, moe_a2a_backend=none) flips mlp_mode to FULL, which makes
NSACPLayerCommunicator gather hidden states across CP ranks before MoE and scatter
back after.
The gather used a raw even attn_cp_all_gather_into_tensor into get_local_dp_buffer(),
and the scatter used tensor_split(cp_size) + reduce_scatter. Both require equal
per-rank shards. Under --enable-nsa-prefill-cp-shared-kv the per-rank shards are
UNEVEN: the page-aligned in-seq split page-rounds each request's 2*cp_size segments,
so per_rank_actual_token differs across ranks. Result: "output tensor size must be
equal to world_size times input tensor size" in the CP all-gather.
Fix: make the round-trip pad-aware, mirroring the NSA index/KV gather idiom that
already handles uneven CP shards via max_rank_len.
- Gather: _cp_attn_tp_all_gather_padded_tensor pads this rank's shard to
max_rank_len and all-gathers into a [max_rank_len * cp_size, H] rank-major
buffer (allocates its own buffer; get_local_dp_buffer()'s length can be smaller
than max_rank_len*cp_size once page-padding inflates the per-rank max).
- Scatter: tensor_split(cp_size) now yields equal max_rank_len chunks (canonical
in-place reduce-scatter layout); reduce-scatter sum-combines this rank's EP-local
MoE partials, then slice to per_rank_actual_token to drop the padding rows,
realigning with the SCATTERED residual.
Strict generalization: when shards are even (non-shared-KV) the padding is zero and
this reduces to the previous behavior. Gated by nsa_use_prefill_cp, so decode/non-CP
and the DeepEP baseline (mlp_mode SCATTERED) are untouched. No global_num_tokens
2*cp_size padding needed: max(per_rank_actual_token)*cp_size == ceil_align(total,
cp_size), which the existing cp_size alignment already provides.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CP draft shared-KV target prefill only needs the CP-local draft hidden side channel until draft prefill consumes it. The reused ModelWorkerBatch was left with capture_draft_hidden_states enabled, and LogitsProcessorOutput retained the extra draft_hidden_states tensor past that point. Restore the capture flags after target prefill, disable draft-hidden capture for draft prefill, and drop the consumed reference.
Constraint: Large CP prefill batches are memory-sensitive; an extra hidden tensor held across the speculative step materially increases peak memory.
Rejected: Leave capture_draft_hidden_states sticky across draft prefill | it can capture or retain hidden state outside the target-side CP-local side channel.
Confidence: high
Scope-risk: narrow
Directive: Keep CP-local draft hidden as a target-prefill-only side channel unless the draft worker explicitly owns a new lifetime contract.
Tested: g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_paged_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/speculative/test_eagle_worker_v2_cp_hidden.py
Not-tested: full speculative integration/E2E workload.
(cherry picked from commit f31ef2293ce0cdda2359a5f4713996d78b47f9df)
The CP shared-KV remap caches may receive tensor views that share the same storage but represent different logical page rows. Keying only by storage pointer and shape can reuse a stale remap for another view, corrupting cache-hit materialization. Use the actual tensor data pointer plus stride, storage offset, and version so different views and mutations are not collapsed into one cache entry.
Constraint: CP shared-KV bs>1 cache-hit paths reuse small tensor views over shared backing tensors.
Rejected: Clear the remap cache on every call | would avoid aliasing but add avoidable hot-path churn and hide the identity bug.
Confidence: high
Scope-risk: narrow
Directive: Do not reduce the remap cache key back to storage pointer only; same-storage views are semantically distinct here.
Tested: g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_paged_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/speculative/test_eagle_worker_v2_cp_hidden.py
Not-tested: local pytest for CP runtime import is blocked by missing starlette in the local base environment.
(cherry picked from commit 7360ef13565dbf21428e3b121112135a2955f913)
Adversarial review of b3a4f4174f found 3 items (Bug B 2b8593ff24 accepted as-is):
1) Re-add the modelopt_quant else-branch input-scale EP-slice (reverting it re-opened a general latent
EP>1 NVFP4 crash on the plain/triton path: --moe-runner-backend triton --quantization modelopt_fp4
--ep>1). No-op at ep==1; orthogonal defense-in-depth (draft now takes the trtllm scalar branch).
2) Draft uses the NON-routed flashinfer_trtllm (FlashInferFP4MoE only does the non-routed kernel; the
_routed impl-class isn't wired). Reject explicit --speculative-moe-runner-backend flashinfer_trtllm_routed.
3) Tighten the checkpoint probe to key on weight_scale_2 (fp4-specific) instead of the generic
weight_scale substring (which also matches fp8 weight_scale_inv).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_set_default_nsa_backends defaulted CP prefill to trtllm on Blackwell+fp8-KV, but the trtllm NSA
prefill path mis-shards k_rope under CP: the rope input carries all CP ranks' tokens (nnz*cp) while
the per-rank rope kernel asserts k_rope_in.size(0)==nnz -> 'Check failed: 256 vs 64' at cp=4. The
function's own comment already notes flashmla_sparse pads heads 64->128 and works under CP; this just
auto-selects it under CP instead of requiring an explicit --nsa-prefill-backend override. Matches the
gov journal (2026-06-08 entry: trtllm-for-CP-prefill was the wrong backend). Decode/non-CP unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ROOT PROBLEM (opus-diagnosed): the NextN/MTP draft MoE is a normal DeepSeek-V3 fp4 MoE, identical in
kind to the main model's — but our fork forced the speculative MoE backend to "auto" (+ a blanket
assert forbidding trtllm-as-speculative), so the draft built under speculative_moe_backend_context →
MOE_RUNNER_BACKEND=AUTO → plain FusedMoE → the non-flashinfer process_weights 'else' branch that
doesn't reconcile EP scale sharding → w13_input_scale(global 256) × w13_weight_scale_2(local 64) crash.
The trtllm-as-speculative ban is over-broad: the Renormalize-only routing constraint is for the BF16
trtllm MoE; the FP4 trtllm kernel supports the DeepSeek routing the NextN carries (FlashInferFP4MoE.
forward_impl). So the elegant fix is to let the draft run the SAME flashinfer_trtllm path as main when
the NextN is fp4 — scoped via the checkpoint's safetensors index (ground truth: do the MTP experts
ship fp4 scales). DeepSeek-R1/V3-FP4 ship a bf16 MTP -> still drop to bf16 + keep the non-trtllm path.
- weight_utils: add shared nextn_moe_is_fp4_quantized_in_checkpoint + load_safetensors_index_weight_map.
- deepseek_nextn: delegate the keep/drop probe to the shared helper (one source of truth).
- server_args._handle_speculative_decoding: for an fp4 NextN, default the speculative backend to the
main trtllm backend (instead of "auto"); narrow the assert to allow trtllm only for an fp4 MTP.
- modelopt_quant: REVERT the else-branch input-scale slice band-aid (de726b4e7e) — no longer hit; the
fp4 draft now takes the trtllm .max() scalar branch, same as main.
Bug B (k_rope 256-vs-64 in the trtllm NSA *prefill* path under CP) is independent and still pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The NextN/EAGLE draft runs under speculative_moe_backend_context -> MOE_RUNNER_BACKEND=AUTO (trtllm is
forbidden as a speculative backend), so its NVFP4 MoE takes the plain-FusedMoE 'else' branch of
process_weights_after_loading. That branch left w13_input_scale/w2_input_scale GLOBAL (num_experts=256,
they are tagged _sglang_require_global_experts at load) while w13_weight_scale_2 is EP-local (64) ->
_compute_gemm1_alphas did 256*64 -> RuntimeError. trtllm/cutlass collapse to a scalar and cutedsl already
slices; the else/triton path was a latent EP>1 NVFP4 bug. Slice the global input scales to this rank's
local experts (mirrors the cutedsl _slice_scale). No-op for moe_ep_size==1; main model unaffected
(never takes this branch). opus-diagnosed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runner-backend heuristic is fundamentally the wrong signal: it reads AUTO in the EAGLE draft
worker (verified live), and an fp4-capable runner does not imply an fp4 MTP — DeepSeek-V3-0324-FP4
runs --moe-runner-backend flashinfer_trtllm yet ships a bf16 MTP (CI test_deepseek_v3_fp4_mtp_small).
is_layer_excluded is also disqualified: R1/V3 leave the MTP unquantized-but-absent-from-`ignore`.
Complete fix (opus-designed): inspect the checkpoint's model.safetensors.index.json and keep fp4 iff
the NextN/MTP experts (model.layers.{num_hidden_layers}.mlp.experts.*weight_scale*) ship fp4 scales —
the ground truth. GLM-5.2-NVFP4 has them (verified: 1536 layer-78 scale keys) -> keep; R1/V3 don't
-> drop. Unreadable index -> drop (PR #7376 default, bf16 MTP fallback, never crashes). Removed the
backend heuristic + debug log + unused get_moe_runner_backend import. Resolves both the 6144-vs-3072
draft crash AND the flood of "model.decoder.mlp.experts.*_scale not found in params_dict" (same root
cause: dropping fp4 left the unquant MTP with no scale params for the checkpoint's fp4 scales).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The NextN/MTP draft is built by the EAGLE draft worker, where get_moe_runner_backend() didn't resolve
to flashinfer_trtllm, so the fp4 drop fired anyway -> 6144-vs-3072 crash in draft load. Check both the
main and speculative runner backends (spec is AUTO when --speculative-moe-runner-backend unset); keep
fp4 if either is a flashinfer fp4 backend. Temporary [NextN-fp4] warning logs the resolved backends.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The team-custom FlashInferFP4MoE.forward_impl read gemm{1,2}_{weights,scales}_fp4_shuffled, but the
HEAD-advanced align_fp4_moe_weights_for_flashinfer_trtllm now stores into w13_weight/w2_weight/
w13_weight_scale/w2_weight_scale (weights uint8, scales already fp8). Rename the 4 reads (Option a:
preserves the team forward incl. its DeepSeekV3 fp32 router_logits cast — GLM-5.2 uses DeepSeekV3
routing, so option-b delegation that drops the cast was avoided). g1_scale_c/g1_alphas/g2_alphas/
w13_input_scale_quant already match HEAD writes (opus-verified 1:1 format/semantics, no silent-garbage).
Cannot wholesale-port layer.py/ep_moe (team EP/CP infra). Watch at launch: draft accept-len + sane output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ported flashinfer_trtllm.py (Stage 1) calls prepare_static_weights_for_trtllm_fp4_moe(is_gated=...),
but utils.py (which DEFINES it) was not ported → old signature → TypeError at process_weights.
utils.py has 0 local edits; wholesale-replace to HEAD. New fn is gated-aware (gemm1_rows), passes
is_gated_act_gemm to flashinfer _maybe_get_cached_w3_w1_permute_indices (verified present on 0.6.12).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GLM-5.2-NVFP4's NextN (MTP) layer-78 routed experts are fp4, but deepseek_nextn unconditionally
dropped modelopt_fp4 for NextN (a DeepSeek-R1-FP4 assumption: its MTP is unquantized). Two fixes
(opus-designed, full back-compat matrix):
A) Drop fp4 for NextN only when the MoE runner can't consume it. Keep fp4 for flashinfer
trtllm/trtllm_routed/cutedsl/cutlass (extends wxiwnd's helper to include our trtllm runner).
R1-FP4/V3-FP4 (non-flashinfer-fp4 or unquantized MTP) still drop → bf16, no regression.
B) The NextN hf_to_sglang_mapper was hardcoded `model.layers.61 -> model.decoder` (R1's MTP idx).
For GLM (MTP=layer 78) this never remapped the layer-78 self_attn/indexer exclude patterns to
the internal model.decoder prefix → NextN attention wrongly quantized fp4 → crash. Worse, keeping
the hardcoded 61 would mis-remap GLM's *real* layer 61. Fix: empty the class mapper and inject
`model.layers.{num_hidden_layers} -> model.decoder` per-config in loader.py (gated on
num_nextn_predict_layers; dataclasses.replace, no class mutation), generalizing to any MTP index.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NSA-prefill CP in-seq-split no longer hard-forces moe_a2a_backend=deepep. Only default to deepep
when the user left both moe_a2a_backend=none AND moe_runner_backend=auto; with an explicit runner
(e.g. flashinfer_trtllm) keep a2a on the standard/allgather path, validated to {deepep,flashinfer,none}.
Adapted from wxiwnd@b300-combact-fix 317acf6c99 (reconciled to our HEAD). Also relax the tp_size==8
assert to single-node tp_size in {4,8} so the GLM-5.2-NVFP4 tp4/CP4 PD target can start; CP code is
parametric in cp_size and the tp%cp / tp%(dp*cp) divisibility asserts still guard validity.
GLM-5.2 (GlmMoeDsaForCausalLM = DeepSeek v3.2 arch) inherits DeepseekV2DecoderLayer, which already
selects NSACPLayerCommunicator under CP -> no GLM-side communicator change needed (decouple stage B
N/A; Glm4MoeDecoderLayer is the unrelated GLM4 arch). Runtime FULL<->SCATTERED verify pending launch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the flashinfer port ("port全"): ADD flashinfer_cutedsl.py (C2, deepep-paired NVFP4 MoE
runner; selectable via --moe-runner-backend flashinfer_cutedsl, already a MoeRunnerBackend enum) and
REPLACE flashinfer_comm_fusion.py to HEAD (C3, allreduce fusion). Both pull self-contained infra
modules target lacked: ADD model_executor/cuda_graph_config.py (cuda_graph_fully_disabled, 0 sglang
top-level imports) and runtime_context.py (get_parallel, lazy imports). comm_fusion consumers
(communicator.py is_flashinfer_allreduce_unavailable, layernorm.py flashinfer_allreduce_residual_rmsnorm)
verified present in HEAD.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two latent CP shared-KV L2 correctness fixes, landed BEFORE L3 (not in an L3
commit). Surfaced by a per-CP8-instance (CP8DP2EP16) scoping review.
PREREQ-1 (CP-group scoping). The B1 commit/evict consensus collectives
(writing_check ReduceOp.MIN, placement_digest MIN/MAX + its tp_world_size<=1
entry guard, drain_storage_control_queues, the evict/prefetch MINs, the flush
barrier) AND cache_controller.prefetch_tp_group all derive from
self.tp_group/self.tp_world_size, which was params.tp_cache_group. tp_cache_group
equals the CP group ONLY at dp_size=1 (enable_dp_attention False -> tp_cpu_group,
and attn_cp_size==tp_size -> _ATTN_CP==_TP) -- the sole reason B1 works today.
Under CP8DP2 (DP attention, attn_tp_size=1) tp_cache_group is the size-1 attn-TP
group, so every `tp_world_size>1` collective silently no-ops per rank and the
placement assert self-disables -> divergent placement -> shared-slab corruption.
Fix: for CP hicache (cp_size>1) scope self.tp_group to the CP cpu group
(get_attention_cp_group().cpu_group -- already used for the slab-handle
broadcast) + self.tp_world_size to its size. A no-op handle change at dp_size=1
(same group object); the intended fix at CP8DP2. The single init-point change
propagates to every consensus collective + un-gates prefetch_tp_group + re-enables
the placement assert.
PREREQ-2 (flush reset). HiRadixCache.reset() cleared the radix tree + host pool
but never reset CpSharedL2PageAllocator -> stale free list/ranges/committed after
flush_cache (leak; the shared pool was never reclaimed). Added
CpSharedL2PageAllocator.reset() (rebuild the per-slab free list all-free, drop
ranges + committed, restore the freshly-built placement_digest) called from
cache_controller.reset() after the ack queues are cleared. Safe: flush_cache is
idle-gated (no in-flight backup/reserve). This is also L3's clear hookpoint.
Validation: new test_reset_restores_freshly_constructed_all_free_state + 89/89
pool suite (torch-2.11 container) + import smoke. PREREQ-1 is a no-op at dp_size=1
(live no-regression confirmed on the next prefill restart); CP8DP2 correctness is
by construction (CP-group membership verified) pending a 2-machine run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Target-model skip-topk layers reuse the previous active layer's top-k indices and should not run local indexer modules. Centralize the layer-needs-indexer decision, skip constructing indexers on shared target layers, and skip their checkpoint tensors during load while keeping nextn/draft layers conservative for state safety.
Constraint: index skip should reduce GPU memory in both prefill and decode without changing top-k propagation semantics
Constraint: nextn/draft layers report shared top-k behavior but still need local indexer state safety
Rejected: Loader-only filtering | parameters are already allocated during model construction
Rejected: Dummy indexer modules for skipped layers | preserves most of the memory cost this change removes
Confidence: high
Scope-risk: moderate
Directive: Do not reintroduce indexer execution on skip_topk target layers without proving prev_topk propagation and weight residency semantics
Tested: remote g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/speculative/test_spec_utils.py test/registered/unit/configs/test_nsa_index_layers.py test/registered/unit/models/test_deepseek_index_skip_weight_loading.py -> 19 passed
Tested: remote g0034 cjy-glm5-new py_compile for modified runtime files
Not-tested: full GLM5 model restart memory delta measurement
EAGLE verification can pass torch scalar tensors through the speculative tree traversal before calling grammar backends. xgrammar/tvm_ffi requires Python int token ids, so normalize traversal indices and draft token ids at the boundary while leaving tensor storage unchanged.
Constraint: xgrammar/tvm_ffi rejects torch scalar tensors for GrammarMatcher.accept_token
Rejected: Coerce tokens inside every grammar backend | the invalid value originates in speculative traversal and should be fixed before backend dispatch
Confidence: high
Scope-risk: narrow
Directive: Keep grammar backend calls scalar-Python typed; do not pass torch scalar tensors through accept_token
Tested: remote g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/speculative/test_spec_utils.py test/registered/unit/configs/test_nsa_index_layers.py test/registered/unit/models/test_deepseek_index_skip_weight_loading.py -> 19 passed
Tested: remote g0034 cjy-glm5-new py_compile for modified runtime files
Not-tested: live decode replay after this commit
Tier-2-GC follow-on (design §9.12 flagged vestige). The metadata's `committed`
bool and `committed_payload_layers` dict were write-only residue of the option-A
quorum the GC removed: set at construction, validated in __post_init__, copied in
split() -- but their only content/True writer was the deleted
_commit_cp_shared_l2_layers_batched. Verified tree-wide there is NO live reader
(`committed` appears only in split's `committed=self.committed` copy;
`committed_payload_layers` only in __post_init__ validation + split copy +
construction). Commit consensus under B1 is the allocator's `_committed_objects`
(MIN frontier), so `metadata.committed` was a redundant second source of truth.
CORRECTION to the original flag: `required_payloads` is NOT a vestige -- it is
LIVE (read by cache_controller.load_cp_shared_l2, the evict path hiradix:2748,
and the readback payload planning). KEPT.
Removed: the 2 dataclass fields + their __post_init__ validation + the two split()
child-copies + the cache_controller reserve-construction kwargs + 4 test
constructions/assertions. No behavior change (fields were never read). Pool suite
88/88 (syh-dev-new/torch-2.11 env) + import smoke confirm the metadata dataclass is
now {logical_len, padded_len, page_size, object_ranges, required_payloads, object_key}.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When --enable-cp-shared-physical-l2-hicache is on, the lane-stats occ/used_tokens
read 0 (they derive from _cp_counts, which the accounting skips for shared-L2
metadata) -- the per-rank gauge is meaningless. Repoint occ/used/cap at the
shared pool's per-payload-slab fill (target_kv/draft_kv/index_k) via the new
CpSharedL2PageAllocator.occupancy_by_payload(); add occ_by={payload,rank} +
occ_labels so the line is self-describing. The real cp_size (owner round-robin
period for the hot-prefix skew) is now computed independently of the occ-vector
length. Per-rank (flag-off) path unchanged. Read-only; rank-0; next-restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CP_HICACHE_LANE_STATS occ/used_tokens are derived from _cp_counts, which the
accounting explicitly skips for shared-L2 metadata (_cp_account_*: `not
_is_cp_shared_l2_metadata`), so they read 0 with l2 pooling on regardless of
actual fill -- wrong gauge for the shared pool. Add cache_controller
.cp_shared_l2_cache_report() (pages_used / pages_capacity / objects_committed /
objects_aborted / objects_evicted + load_hits/misses from the replicated
CpSharedL2PageAllocator stats) to the lane-stats line as `shared_l2_pool=...`
when the allocator is constructed, so fill->evict->load-back is observable.
Read-only; rank-0 only; takes effect on next restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flag-on 2b.3 bring-up (--enable-cp-shared-physical-l2-hicache, real GLM-5.1
PD traffic) surfaced two partial-adopt gaps from the piecemeal 2b.0/2b.1b ports
from l2_pooling — a caller brought in without its callee/declaration. Neither
was caught by the pool unit tests or import smoke (they don't exercise the
device-allocator load path or the flag-on capacity-snapshot path); only live
flag-on traffic hit them.
1. environ.py: declare SGLANG_CP_HICACHE_VERIFY_SNAPSHOT (EnvBool, default False).
hiradix _cp_host_capacity_snapshot (:1477) reads it to gate a fail-loud
running-count drift verifier (_cp_walk_capacity_counts vs _cp_counts), but the
env field was never declared -> AttributeError crash on the first flag-on
scheduler tick. The verifier IS fully implemented (it's the count-machinery
analog of the placement_digest assert) -- keep it, default-off (expensive
full-walk; on in bring-up/CI).
2. allocator.py: restore alloc_pages_for_shared_l2_load (thin wrapper over the
adopted alloc_pages_with_owners) on the CP shared paged allocator. 2b.0 ported
the callee but dropped this wrapper that cache_controller.load_cp_shared_l2
calls -> RuntimeError on every L2->L1 cache-hit reload until restored.
Validated live: prefill boots with the flag, shared-L2 slab constructs over
hugetlbfs on all 8 ranks, write-through + load-back + cache hits + coherent
output all work, placement_digest assert green (no divergence), no crash.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the 2b.1b shared-L2 capacity-miss SKIP with a deterministic
collective-free evict-to-fit, and deletes the #5 deadlock. This is the gate that
makes --enable-cp-shared-physical-l2-hicache safe in a real run.
evict-to-fit (_reserve_write_cp_shared_l2_evict_to_fit, hiradix): on a shared-L2
reserve capacity miss -- which raises identically on every CP rank over the
replicated free list -- snapshot the plannable shared-L2 host leaves (excluding
the node being backed up) in the replicated SLRU order (_cp_host_evict_key =
Phase-1 logical clock + node.id total-order tiebreak), release the coldest one at
a time (each a replicated free-list mutation via the proven
_evict_cp_host_for_write_admission teardown -> evict_cp_host -> allocator.release)
and retry the pooled reserve after each, stopping at the first fit.
Fragmentation-correct (free coalesces in _return_range). Identical victim
set + order + deterministic reserve => every rank evicts the same victims and
stops at the same point (design Thm 1), so NO collective is needed. The snapshot
is one-level by design (a parent promoted to a leaf mid-loop is not re-pushed;
deep cascades skip-and-retry next tick -- missed identically on all ranks).
Exhaustion -> loud rate-limited skip (transient: pinned/in-flight objects hold
the pool), never a hang.
#5 deadlock DELETED: stripped the synchronize_across_ranks all_reduce machinery
(the `while len(heap) and not all_ranks_done()` ReduceOp.MIN host_evict_done_min
closure + the param) from _evict_host_for_physical_slots -- never reached (no
caller passed True) and the headline CP-deadlock shape. evict_host(num_tokens)
still works (single-arg, equivalent non-sync loop).
Opus adversarial review = SHIP (all 7 findings PASS: determinism [_cp_host_evict_key
purely replicated, pin_expiry dormant under CP], teardown byte-equivalence + no
double-free, no infinite recursion + clean abort-on-partial, snapshot safety [no
ancestor/descendant among host leaves], bounded termination, clean #5 strip,
flag-off path untouched). Tests: new test_eight_ranks_evict_to_fit_stays_identical
(fill->miss->release-in-order->retry -> identical placement, exercises coalescing)
+ 88/88 pool suite (syh-dev-new) + import smoke.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2b.1b adopt-then-strip left the entire option-A commit-quorum apparatus
orphaned: B1's writing_check ReduceOp.MIN frontier (mark_object_committed) is
the all-ranks-done consensus, so the per-(rank,layer,payload) gather quorum it
replaced is information-redundant (design sec 2.4 Corollary). This GC removes
the apparatus and collapses the CpSharedL2PageAllocator commit model to exactly
{_ranges_by_object, _committed_objects, _free_by_payload} -- precisely what
placement_digest() hashes -- so the cross-rank digest assert now covers the
whole model and the dead quorum is structurally unrepresentable.
Removed (all proven dead under B1, confirmed by a full tree caller-sweep +
opus adversarial review = SHIP):
- allocator: commit_layer, _has_full_commit, _expected_layers_for_object_payload,
adopt_reserved_range, set_object_required_payloads; fields _commits_by_object,
_expected_ranks, _adopted_ranges, _required_payloads_by_object,
_expected_layers_by_object, _required_payloads, _expected_layers; ctor args
expected_ranks/expected_layers/required_payloads; module fns
broadcast_cp_shared_l2_decision, gather_cp_shared_l2_{preflight,commits,commit}.
reserve/split_committed_object/_drop_object simplified to ranges+committed-bit.
- cache_controller: the 3 dead _commit_cp_shared_l2_* fns, 3 imports, all 5
option-A ctor params (cp_shared_l2_{cpu_group,source_rank,broadcast_fn,
preflight_fn,commit_fn}), the per-object commit-contract builder/setter/call.
- hiradix: allocator construction updated (3 args dropped); live _split_node
caller and _get_cp_shared_l2_rank_and_group@167 intact.
- tests: split tests migrated to mark_object_committed; 4 dead-symbol tests removed.
Lost __init__ payload-has-slab validation is covered by reserve()'s own
ValueError (fail-loud at first write). Default (flag-off) path unchanged: the
allocator is never constructed when enable_cp_shared_physical_l2_hicache=False.
Net -580 LOC. Validated: 87/87 pool suite (syh-dev-new, torch) + 31/31 core
classes locally + edited-module import smoke. Out of scope (flagged separately):
the CpSharedL2NodeMetadata.{required_payloads,committed_payload_layers} vestige.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The B1 write-through: every CP rank runs the IDENTICAL deterministic CpSharedL2PageAllocator.reserve over the replicated event stream -> identical placement with NO broadcast/gather; commit rides the writing_check ReduceOp.MIN frontier (mark_object_committed) instead of a per-(rank,layer,payload) gather quorum.
cache_controller.py: _reserve_write_cp_shared_l2 stripped to every-rank reserve (dropped preflight#2/rank0-gate/broadcast#3/adopt/missing-payloads; per-object contract kept MF3; idempotent abort-on-partial SF3); _submit_write_cp_layer_states drops the 3 gather-commit calls (per-layer D2H + all-payload done-gate MF1 + per-node ack kept); submit_write_cp_all_layer drops both fallback gathers (zero-owned keeps its ack MF2).
hiradix_cache.py: 3-way merge of l2_pooling shared-L2 init (clean, 0-conflict; Phase-1 clock + E1 + reserve-budget preserved) builds the allocator/slab when the flag is on and passes it to HiCacheController WITHOUT collective fns. The #5 deadlock removed: shared_l2_capacity reserve-miss now SKIPS the backup (reactive, rank-uniform) instead of _evict_host_for_physical_slots(synchronize_across_ranks=True) -- the deterministic shared-pool evict is 2b.2. mark_object_committed wired at _commit_pending_backup (the MIN commit). New _cp_assert_placement_replicated (SGLANG_CP_HICACHE_PLACEMENT_ASSERT) in check_hicache_events: MIN/MAX-reduce placement_digest across the CP group, fail-loud on divergence (Theorem 1 runtime gate; rank-uniform entry, cannot deadlock).
environ.py: SGLANG_CP_HICACHE_PLACEMENT_ASSERT (EnvBool, default off).
Tests: 8-rank reserve-determinism (identical placement across 8 ranks + divergence-detectable) + full pool suite 91/91 + import smoke on g0033 syh-dev-new. TWO opus adversarial reviews both SHIP (cache_controller strip + hiradix wiring).
NOTES: default-off (--enable-cp-shared-physical-l2-hicache); validated for write_through policy (prod default). Capacity-miss currently SKIPS (B1 shared-pool evict = 2b.2) so the flag must stay off in real runs until 2b.2. Dead gather island (3 fns + 3 imports, opus-confirmed inert) pending GC follow-up. Design+proof: docs_internal/cp_hicache_2b_pooled_l2_b1_design.md sec 2.4/9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CpSharedL2PageAllocator.mark_object_committed(object_key) -- the B1 commit path that marks an object committed directly, bypassing the per-(rank,layer,payload) commit_layer/_has_full_commit quorum. Under B1 the all-ranks-done consensus is the writing_check ReduceOp.MIN frontier (the same barrier that releases the node write lock), so every rank calls this with the same object_key at the MIN commit point and the committed set transitions rank-uniformly (feeds placement_digest).
Resolves opus-review MF4: split_committed_object stays coherent for a B1-committed object (no per-layer commit facts) -- it adds children to _committed_objects and the empty _commits_by_object is harmless because B1 never calls _has_full_commit.
7 unit tests (pure-Python): commit-without-quorum, idempotent, unknown-raises, stat bump, digest reflects committed set + lockstep equality, split-of-marked-object coherent, release-after-mark frees pages. Full suite 88/88 green on g0033.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CpSharedL2PageAllocator.placement_digest() -- a deterministic, order-independent SHA-256 of the full replicated allocator state (free list + per-object ranges + committed set). This is the B1 proof-obligation gate from the 2b design (sec 2.4, Theorem 1): the write-through (2b.1) cross-rank assert hashes this each tick and EQ-checks it across CP ranks, turning placement-determinism into a fail-loud runtime invariant -- any reserve/release fired on a subset of ranks or with a per-rank arg changes the digest.
6 unit tests (pure-Python, no torch): determinism, changes-on-reserve, identical-op-sequence => identical-digest, divergent-op => different-digest, reserve+release restores digest (free-list coalescing), placement-order sensitivity. Full suite 81/81 green on g0033 syh-dev-new.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopt the ownerless shared-physical-L2 mechanics from origin/l2_pooling onto the Phase-1 branch, behind --enable-cp-shared-physical-l2-hicache (default off -> behavior-neutral). Foundation for the B1 collective-free allocator (2b.1/2b.2) and the L3 durable floor (Phase 3); no coordination yet.
- cp_shared_l2_pool.py (new): CpSharedL2PageAllocator (deterministic first-fit, capacity charged once, no x cp_size), hugetlbfs-2M slab primitives, position-indexed object ranges, commit-quorum data model.
- memory_pool_host.py: SharedHostTensor* slab orchestration (3-way merged; fix-output's net change to this file is blank-line-only -> additions-only, no content lost).
- server_args.py: flags + _handle_cp_shared_physical_l2_hicache_validation (3-way merged clean with Phase-1 assert + fix-output EnvField).
- test_cp_shared_l2_pool.py: 75 unit tests (allocator/slab/quorum), green on g0033 syh-dev-new; envs stub provides SGLANG_REQ_WAITING_TIMEOUT (read by Phase-1 validation).
Design + B1 proof: docs_internal/cp_hicache_2b_pooled_l2_b1_design.md (sec 2.4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make CP shared-KV HiCache eviction collective-free and scan-resistant by
removing every per-rank wall-clock input from the eviction/admission decisions
(any such input desyncs the must-be-replicated victim set / batch across the 8
CP ranks and can deadlock the collective-coupled writeback/load-back).
- Replicated logical clock: TreeNode.next_access_time() (a process-global
monotonic counter, mirrors mamba/swa's get_last_access_time) replaces
time.monotonic() as the source of last_access_time at every radix bump site
(radix_cache __init__/match/insert; hiradix CP match/insert/_insert_helper_host;
reset() zeroes it). creation_time/pin_expiry intentionally stay wall-clock.
Because match/insert events are replicated (reqs broadcast from rank 0 over the
replicated tree), last_access_time is now identical on every rank. Unique ints
also give a strict total order (no LRU tie ambiguity). No duration arithmetic
reads last_access_time, so non-CP LRU is unchanged; mamba/swa use their own
TreeNode and are untouched.
- CpReplicatedSLRUStrategy = (is_protected[hit>=2], last_access_time, node.id):
scan-resistant (cold one-shots stay probationary, evicted before reused/
protected prefixes), recency-within-segment ages out cold (not LFU), node.id
is the deterministic total-order tiebreak (heaps never compare TreeNodes).
Overridden for CP so it reaches every get_priority eviction site; the host
write-admission key _cp_host_evict_key is switched from FIFO-by-id to it.
- Co-fixes for the same per-rank-wall-clock class: pin_prefix is forbidden under
CP (its pin_expiry victim-eligibility check is per-rank wall-clock; SLRU
scan-resistance covers the benefit); the affinity head_age_s admission input is
disabled (=0.0, relying on the replicated head_defer_count bound); and a
server_args fail-fast guards CP shared-KV against SGLANG_REQ_WAITING_TIMEOUT>0
(its waiting-queue pruning is per-rank wall-clock).
Tests: test_evict_policy.py adds TestCpReplicatedSLRUStrategy (scan-resistance,
recency, total-order, full ordering) -- 30 passed in the dev-cu13 container;
test_radix_cache_unit.py adds logical-clock determinism/total-order tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
