Rank-local CP HiCache async write queues can drain at different times across ranks. Radix structural decisions that consult those local queues can diverge, so backup-pending state now lives on the replicated radix node and uses a separate backup lock ref from request locks. Commit and rollback clear the replicated marker, and split/prune/hit-count probes use the marker rather than local pending dictionaries.
Constraint: CP ranks must make identical radix structural decisions even when local async write acknowledgements drain at different times
Rejected: Drain write acknowledgements before every split/prune decision | still rank-local and adds hot-path synchronization
Rejected: Treat backup lock_ref as a normal request lock | hides the distinction between request protection and backup-pending protection
Confidence: high
Scope-risk: moderate
Directive: Do not derive CP HiCache split/prune/write-through decisions from pending_host_backups or ongoing_write_through without a replicated node marker
Tested: local py_compile python/sglang/srt/mem_cache/hiradix_cache.py python/sglang/srt/mem_cache/radix_cache.py
Tested: local git diff --check
Tested: remote cjy-glm5-new pytest replicated marker test plus targeted CP HiCache metadata subset, 10 passed
Not-tested: full test_cp_hicache_metadata.py because current branch has unrelated fixture drift around enable_cp_l3/cp_shared_l2_page_allocator/running-count setup
Co-authored-by: OmX <omx@oh-my-codex.dev>
Drives _cp_l3_reserve_reloads directly against a real CpSharedL2PageAllocator:
- same-tick state (l3rl object committed-and-live + node attached -> attach_ok False)
no longer raises 'duplicate live reservation'; the candidate is dropped and the
request left un-held (re-matches the inserted node).
- WITHOUT the recheck (attach_ok forced True = pre-fix path) the identical state
raises the duplicate error -> proves the recheck is load-bearing.
- an in-flight reload still piggybacks (recheck sits after the dedup, not before).
- a genuine clean miss still reserves (no false-positive starvation).
CPU unit test (register_cpu_ci, stage-a-test-cpu); runs in the dev-cu13 container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the O(victims*candidates) per-iteration greedy argmin rescan in
_plan_cp_load_back_owner_lane_evictions with a leaf-up min-heap. The score
(-contribution, -unlock, slru_priority, node.id) depends on the current deficits, so a
static heap is not equivalent; instead use LAZY RE-EVALUATION: a popped entry is consumed
only if its score still matches the node's current score (deficits unchanged since push),
else it is re-pushed with the fresh score. Stale scores are always optimistic (contribution
= sum(min(counts[o], deficits[o])) and the ancestor-unlock contribution only shrink as
deficits shrink), so the first up-to-date popped entry is exactly the global argmin the full
rescan would have picked -> PROVABLY EQUIVALENT, with the same leaf-up eligibility +
ancestor-unlock + parent-push. Determinism/rank-uniformity preserved (selection decided by
the total-ordered score; the heap insertion seq only orders structurally-equal tuples).
Equivalence proven by test: a verbatim reference greedy + a randomized property test (300
seeds, single- AND multi-owner deficits, non-uniform counts exercising the lazy-re-eval
boundary, varied SLRU priorities) asserting byte-identical (victims, planned_freed,
remaining), plus an explicit leaf-up + ancestor-unlock case (child-then-parent). 14 planner
tests pass (1 pre-existing unrelated EAGLE-tail failure unchanged).
Micro-bench (V100, candidates=2000 deficit=7877, benchmark/hicache/bench_cp_owner_lane_planner.py):
A. ORIGINAL (.item() x cp, no memo) 50.06s
B. memo + bincount (greedy) 0.494s (101x)
C. lazy-re-eval heap 0.162s (309x; 3.0x over B) selection A==B==C identical
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A CP4 prefill server hung 79s in _plan_cp_load_back_owner_lane_evictions (the [HiCache-load]
slow-scan) and was killed by the detokenizer health-check. Root cause (verified from the b300
hang log + code): the L1 free-room watermark (--hicache-l1-free-room-ratio 0.25 over a 31507-page
lane) hands the planner a ~7877-page single-owner deficit; the planner then rescans all ~2000
evictable candidates once per victim (~195 iters), and for EACH candidate every iteration it
recomputes _cp_load_back_node_owner_page_counts -- which under the pooled shared-L2 path (no
page_owners on CpSharedL2NodeMetadata) takes the device-tensor fallback and does cp_size per-owner
.item() device->host syncs. That is ~195 * 2000 * 4 ~= 1.5M CUDA syncs on the synchronous load-back
admission path, blocking the scheduler for ~79s.
Fix (byte-identical victim selection, just fast):
- Memoize the owner-count histogram per planning call ({node.id: counts}); the counts are invariant
while node.value is fixed (the plan does not mutate values), so node.id is a safe key for the plan's
duration. Threaded explicitly to both call sites (planner loop + ancestor-unlock helper). Turns
O(victims*candidates) recomputes into O(distinct nodes).
- Replace the cp_size per-owner sum().item() loop with one bincount().tolist() device->host sync.
Net: ~79s -> ~1s; the eviction plan (victims, planned_freed) is unchanged. bincount == the per-owner
loop proven over 8000 random vectors incl. the (-1)%cp==cp-1 zero-loc edge. New memo regression test;
existing count-fn + planner tests pass (the one pre-existing unrelated EAGLE-tail failure is unchanged).
(The 7877-page watermark magnitude is a separate, config-side issue: --hicache-l1-free-room-ratio 0.25
reserves ~25% of a ~2M-token lane -- ~30x more than a 64K chunk needs; lower it.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On restart the LMDB index + disk-slab blobs persist, but CpL3Store.from_config built the slot
pools all-free and the GC LRU empty -> the next spill re-hands-out a slot the durable index still
references (clobbers a live blob) and GC never reclaims the carried-over entries. Neither a clean
start nor a durable reload was actually realized.
connect() now applies a cold-start policy BEFORE the bg threads start (the write thread is the sole
pool/GC owner, so the single-threaded reconcile must precede it):
- clear (default): wipe the persisted index + reset the pools/GC -> genuinely empty start (disk
blobs are inert, overwritten lazily on slot reuse).
- load: rebuild this rank's slot free-list + GC LRU from the durable disk blobs. Drive the scan
from the rank's OWN slab file (header-only reads) so it never inspects another rank's slots even
when ranks share a disk; a slot is LIVE iff its blob header parses AND the shared index still maps
that content hash back to this exact slot -> occupy + seed the GC LRU with the durable last_access;
orphan/unwritten slots stay free. Header-only (no payload CRC); reload-time verify-on-read still
fail-softs a torn payload. The L3 durable floor now survives a process restart.
Primitives: CpL3SlotPool.rebuild_from_allocated (O(num_slots) bulk-occupy) + CpL3DiskSlab.read_header
(one aligned block, not the multi-MB slot). Env SGLANG_CP_L3_COLD_START (default "clear"), read in
_maybe_init_cp_l3 and passed to connect(). Tests: 4 cold-start e2e (load rebuilds the floor + no slot
collision after a fresh spill; GC LRU rebuilt; clear starts empty; unknown mode fails loud) + 2 unit
(rebuild_from_allocated, read_header). 23 L3 store/disk/posix tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A radix split crashed: split_committed_object got split_pages=172 (from the node
KEY length) for a target_kv object of num_pages=1 (the host backup), raising
"split page count must leave at least one page on each side". Root cause: the L3
reload inserts a MALFORMED node -- _cp_l3_reserve_reloads stores the FULL n_full-
page suffix key but only the C L3-durable pages are reserved/hashed/host-backed
(C < n_full when L3 holds a partial durable prefix via page-level GC or partial
spill). The node then has len(key)//page = n_full but host_len//page =
object.num_pages = len(hash_value) = C, violating the radix invariant every
consumer assumes; the split derives split_pages from the long key and overruns the
C-page object. Pre-existing (L3 3.2, 864b1c808e); NOT the multi-slab change
(_allocate_contiguous raises on no-fit, never partial, so multi-slab cannot produce
a partial-backed node -- it crashes single-slab too).
Elegant fix -- restore the invariant at its single source: slice the reloaded node
to its C durable pages so len(key)//page == host_len//page == object.num_pages ==
len(hash_value) == C. The un-durable suffix [C:n_full] is a correct cache miss.
This makes _split_node / CpSharedL2NodeMetadata.split / split_committed_object / the
L3 page-set builders correct unchanged, and removes the latent negative-host_len and
value/hash mis-slice the same malformed node would have triggered.
- hiradix_cache.py _cp_l3_reserve_reloads: slice suffix_key to C pages (matching the
already-sliced suffix_hashes).
- hiradix_cache.py _cp_l3_build_owned_pages (spill) + _cp_l3_build_reload_owned_pages
(reload): fail loud if the page-hash count exceeds the object's pages (else a future
desync silently corrupts a neighbour object in the same slab -- the slab accessor
only catches a whole-slab overrun).
- hiradix_cache.py _split_node: assert split_len <= host_len (fail loud on any future
malformed partial-backed node instead of the cryptic split_pages>=num_pages).
- test_cp_shared_l2_pool.py: lock the split contract (split_committed_object rejects
split_pages outside (0,num_pages); metadata.split rejects object/padded mismatch).
Design + 3-agent investigation + 2-agent review (no bugs):
docs_internal/cp_hicache_l3_reload_partial_prefix_fix_design.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A single cudaHostRegister over a >~1 TB host buffer fails with
cudaErrorMemoryAllocation on B300 (hard per-call ceiling: 512 GiB OK, 1024 GiB
FAIL), crashing the hicache_size=1600 (~1.5 TB CP shared-L2 slab) prefill at
startup. The registration cannot simply be chunked: a memcpy (cudaMemcpyBatchAsync,
the CP-L2 H2D/D2H transfer) fails with cudaErrorInvalidValue when its host range
straddles a registration boundary (verified empirically on b300-049).
Fix: physically split the host cache into multiple page-aligned slabs, each <= a
safe single-registration size (default 480 GiB, env SGLANG_CP_HICACHE_MAX_SLAB_GB),
reusing the existing SharedHostTensorGroupAllocator + per-slab transfer splitting
(_host_transfer_segments). Each slab is one whole registration and no transfer
crosses a boundary; small configs (hicache_size<=400) stay single-slab unchanged.
- memory_pool_host.py: add cp_hicache_max_single_register_bytes() + the fail-loud
_check_single_cuda_host_register_size guard; revert the (transfer-unsafe)
registration chunking back to one cudaHostRegister per buffer/slab.
- hiradix_cache.py: _cp_shared_l2_slab_pages_by_payload auto-caps each payload's
slab <= the ceiling so large caches auto-split.
- cp_l3_slab_accessor.py: CpSharedL2SlabAccessor is now slab-count-aware
(CpL3SlabSpan + per-slab dispatch via global_base_page; per-slab layer stride);
_cp_l3_slab_spans rewires _maybe_init_cp_l3 off the single-slab assumption. L3
disk slabs / slot pool / LMDB index / GC are content-addressed and unchanged.
Tests: multi-slab accessor incl. a non-circular torch.frombuffer-layout check; a
real allocate_group + _cp_l3_slab_spans roundtrip; slab-cap auto-split; L3 store
cross-slab spill/reload. Reviewed by 3 adversarial agents, no correctness bugs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The host-pool size check used psutil.virtual_memory().available -- regular RAM -- unconditionally. But
hugepages are carved OUT of regular RAM, so reserving e.g. 1 TB of 2M hugepages drops psutil.available
by ~1 TB, and the check then spuriously fails ("Not enough host memory ... only have 897 GB free") even
though the slab is allocated from the hugepages, not regular RAM. It was inspecting the wrong pool.
Two fixes:
- memory_pool_host.py: gate the psutil check on the DEFAULT (regular-malloc) allocator. A custom
host_tensor_allocator (the CP shared-L2 slab over hugetlbfs) owns its own capacity check, so skip the
regular-RAM check for it.
- cp_shared_l2_pool.py: add cp_shared_l2_hugetlbfs_free_bytes() + check_cp_shared_l2_hugetlbfs_capacity()
(statvfs the hugetlbfs mount: f_frsize = hugepage size, f_bavail = free hugepages) and call it in
create_cp_shared_host_slab before ftruncate/mmap. Now insufficient hugepages fail LOUD with a clean,
actionable message ("insufficient hugepages in /dev/hugepages: need X GB (N x 2M pages) but only Y GB
free; reserve more or reduce --hicache-size") instead of a cryptic mmap ENOMEM. Per-slab + sequential,
so each payload's slab sees the free remaining after the prior ones.
Unit-tested (capacity check: tiny fits, over-capacity fails loud). Pre-existing 2b L2-pooling code (not L3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under CP, enable_storage=False, so StorageMetricsCollector (prefetch/backup tokens + bandwidth) never
emits, and L2 pool usage was debug-log-only (_cp_maybe_log_lane_stats) while L3 was invisible. Add a
CpHiCacheMetricsCollector covering both, gated on enable_metrics alone, per-rank via the tp_rank label.
- CpL3Store.stats(): per-payload slot occupancy (used,total) + cumulative counters (spill_pages,
reload_pages, gc_reclaimed, write_errors -- incremented on the bg threads) + instantaneous inflight
(spill/reload) + the four bg queue depths. Counters are monotonic (NOT reset on clear()).
- CpHiCacheMetricsCollector (metrics_collector.py): gauges for instantaneous state
(sglang:cp_l2_used/total_pages, cp_l3_used/total_slots, cp_l3_inflight_*, cp_l3_queue_depth) +
counters for cumulative work (cp_l2_objects_committed/evicted_total, cp_l3_spill/reload_pages_total,
cp_l3_gc_reclaimed_total, cp_l3_write_errors_total). Counters fed via a monotonic .inc(delta) off the
store's cumulative totals.
- hiradix: create the collector in _maybe_init_cp_l3 (once, per-rank labels from cache_controller);
push periodically (~1s, per-rank wall-clock -- metrics need no rank-uniformity) from
check_hicache_events via _cp_maybe_collect_metrics, reading allocator.occupancy_by_payload()/stats()
+ store.stats(). Off the data path.
store.stats() unit-tested (occupancy + spill/reload counters; 10 store tests green). The collector +
wiring are correct-by-pattern (mirror StorageMetricsCollector) and validate on the live server (3.4;
prometheus_client not in the unit env). Directly useful for 3.4: watch L2/L3 fill + GC + reload hits
under cachebench.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces §9.3's object-level rank-uniform eviction with per-rank, page-level LRU GC running ON THE
WRITE THREAD (the sole slot-pool owner -> no lock, no second owner). Genuinely communication-free:
each (hash,payload) is written/read/deleted by exactly its one owner (owner = i%cp_size, and the
prefix-chained content hash makes one-hash = one-position = one-owner), so the shared LMDB's own
consistency substitutes for collectives within L3 (design §9.5/§9.6). GC touches only L3, adds zero
collectives; the spill/reload MINs that remain are all the L2 bridge.
cp_l3_store.py:
- Per-payload lazy-deletion min-heap `_gc_heap` of (last_access, seq, hash) + authoritative
`_gc_current{hash->(last_access,slot)}`. Add on write, touch via `_touch_q`, reclaim coldest.
- The write loop, each iteration: drain touches -> _gc_collect (reclaim coldest of any pool over the
0.90 start watermark down to 0.85, bounded budget) BEFORE the next write, so spill never hits a full
pool (no evict burst on the critical path). Continuous + proactive, symmetric with the spill.
- Reclaim deletes the index entry (durable) BEFORE freeing the slot, so a racing reload reads None or a
CRC-mismatched reused slot -> miss -> recompute (fail-soft); no lock / in-flight-exclusion needed.
- submit_touch (scheduler->write thread); clear() resets the GC structures.
hiradix_cache.py: wire the reload `touch` (the 3.2 gap -- hit_count/last_access never updated): at
_cp_l3_admit_reload, bump the L3 last_access of THIS rank's owned reloaded pages (ALL pages, not just
the tail, so a prefix ages uniformly). _cp_l3_insert_reloaded_node now returns the node (its
last_access seeds the touch).
Sharing handled optimally for free (page = one entry, last_access = max over touchers; prefix pages
age together; shared-hot survives) -- no object-record table, no refcount, no content-keying. 56 L3
unit tests green (new: GC reclaims coldest-to-watermark + no slot leak; touch keeps a page warm).
Cold-rebuild (scan slab headers + LMDB) is designed (§9.6) but deferred to 3.4 (crash-recovery).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a radix miss, reload the evicted-but-L3-durable suffix from disk into L2 + insert a fresh
radix node, holding the triggering request until then (mirrors the storage-prefetch hold, which
is disabled under CP). The reloaded prefix re-enters the radix as a normal L2 node, so the
existing load_back serves L2->L1 -- no new device path.
Design (docs_internal/cp_hicache_l3_phase3_impl_design.md §9, refined for minimal sync):
- ENTRY (scheduler _prefetch_kvcache, enable_cp_l3 branch): mark a miss-with-suffix as a reload
candidate (rank-uniform: requests are broadcast). cp_l3_reload_lookup computes the suffix's
full-page content hashes -- the SAME SHA chain spill wrote (compute_node_hash_values).
- RESERVE + ADMIT fold into the existing per-tick _drain_l3_control_queues MINs -- ZERO new
collectives: the candidates' per-rank exists_prefix counts ride the qsize MIN vector -> a
rank-uniform agreed count C (this reconciles the async-LMDB read-skew); reload-ack oks ride the
durable ok-AND. Reserve is then collective-free (deterministic evict-to-fit mirroring
_reserve_write_cp_shared_l2_evict_to_fit); the request waits non-blocking in waiting_queue
(check_cp_l3_reload_progress skip) and is admitted + re-matched the SAME tick the reload lands
(check_hicache_events runs before batch formation). CP-aware insert only on a still-clean attach
(else abort+recompute). Capped (cp_l3_reload_max_inflight) + content-key deduped (piggyback).
EAGLE/bigram (opus-studied, PROVABLY correct -- not deferred): the radix key is bigram-converted
over the whole request, and convert_to_bigram_key(fill_ids[M:]) == bigrams(fill_ids)[M:] exactly
(the boundary bigram belongs to the matched prefix), so the bigram-converted suffix hashes
reproduce node.hash_value. Verified vs source + a new slice-identity regression test.
Opus review (FIX-THEN-SHIP, no CRITICAL; the 4 crash surfaces -- MIN-vector shape, read-skew,
attach-anchor, placement-digest -- traced clean) + independently re-verified; fixes folded:
- M1: cp_l3_release_request clears reload state on request abort/timeout/preempt (no leak).
- H1: rank-uniform per-op TTL bound releases a held request if its reload op never acks (the
default SGLANG_REQ_WAITING_TIMEOUT is off); the reservation frees safely at the (late) ack.
- L1: fail-soft anchor guard at insert (re-derive the first suffix hash from lhn's live hash).
- empty-waiters reload aborts (frees L2) instead of inserting an unrequested node.
Imports hoisted to top-level (get_hash_str -- hicache_storage is a leaf module, no cycle;
convert_to_bigram_key already top-level), no inline imports. py_compile clean; 54 L3 unit tests
green. Reload triggering (submit_reload/exists_prefix/free_object) now wired (was write-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first 3.1 (d08ee9f8d) spilled the COLDEST resident objects and held the
protect_host eviction-pin through the entire slow disk write (released only at the
durable ack). Since the coldest objects ARE eviction's victims, a B300 cachebench
wire test wedged: evict-to-fit found everything pinned (evicted=0) ->
host_reservation_failed flood -> caching broke.
Redesign (docs_internal/cp_hicache_l3_phase3_impl_design.md §R3):
- cp_l3_store: split the single spill thread into a GATHER -> WRITE pipeline. Gather
copies each owned page off the pinned slab into a slab-independent anonymous-mmap
staging buffer (fast RAM memcpy) and acks GATHER; a separate write thread does the
O_DIRECT write + fdatasync + durable LMDB put off the staged copy and acks DURABLE.
The eviction-pin releases at the GATHER ack, so the object is evictable right after
the RAM copy; the disk write completes off the staged copy even if L2 reuses the
pages. Slab-full frees its partial slot allocation (no orphan) + acks ok=False.
- hiradix: continuous PROACTIVE spill — enqueue each just-committed object to a FIFO
deque at the replicated commit frontier (_commit_pending_backup) AND at radix splits
(the freshly-committed parent half), drained in commit order (not coldest). The deque
is capped (rank-uniform) so a disk stall can't OOM it. Spilling HOT just-committed
objects (disjoint from eviction's COLD victims) means the cold tail is already
L3-durable when evicted -> eviction just drops it, never contends for the pin.
- 3-element CP-group MIN drain [gather, durable, reload]: gather-MIN -> release_host;
durable-MIN -> l3_durable, gated by an ok-AND (a second MIN over per-op ok bits) so a
write failure on any rank means NO rank marks it durable (rank-uniform durability,
placement_digest stays green).
opus-reviewed (FIX-THEN-SHIP) + independently re-verified; review fixes folded:
split-enqueue (HIGH), deque cap for the disk-stall OOM (MED), fail-soft-on-write-failure
documented (HIGH; deliberately NOT re-enqueued — re-gather would churn under slab-full,
recompute is correctness-safe), rate-limited error logs (MED/LOW). 54 L3 unit tests
green (new: 2-phase gather-before-durable ordering, slab-full fail-soft + no slot leak).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ties config -> per-rank disk slabs + slot pools + shared LMDB index + slab accessors. Background
spill/reload threads (off the scheduler tick, storage-template shape); object-granular acks (one per
object after all owned pages durable; zero-owned ranks ack in lockstep) via ack queues + has_inflight,
so the caller's CP-cpu-group MIN-drain frees the same objects on every rank. Spill = gather->O_DIRECT
write->durable index (data->fdatasync->index-commit ordering) with content-hash dedup; reload =
index lookup->read_into->scatter (verify-on-read). free_object frees owned slots+index (3.3 eviction
consumes); clear() is the flush_cache hook (stop/drain/reset/restart). from_config splits the disk
budget across ranks-on-disk x payloads (equal slot count). PLP gate at connect(). Model B: store is
content-addressed + rank-local I/O; only the LMDB is shared. End-to-end test (spill/exists/evict/reload
byte-exact/dedup/free/clear) 5/5 (venv lmdb). + accessor n_layers/page_num accessors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CpSharedL2SlabAccessor (HIGH-1): memcpy-gather a page's N_layers strided slab slices into a
contiguous buffer (spill) + scatter a contiguous blob back into the strided slices (reload) —
the cheap RAM bridge that lets host stay layer_page_first (inference perf) while disk is
page-contiguous. Generic CpL3SlabLayout(n_layers, page_num, slice_bytes) with verified-from-source
factories: for_mla (layer_num,page_num,page_size,kv_cache_dim,itemsize -> 2.74 MiB/page) and
for_index (n_active_layers,indexer_page_num,indexer_page_stride -> 0.17/0.63 MiB/page). gather_into/
scatter_from take an offset so the spill writes straight after the 64B slot header (zero extra copy).
Pure (mmap+ints), no torch. 8/8 tests. Canonical layer 0..N-1 order shared by spill+reload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Preallocated disk-slab file per (payload_kind, drive); a page = one contiguous blob
(64B header + gathered payload + pad) in a 4K-aligned slot. O_DIRECT single-transfer
write/read (measured best vs vectored), buffered+fdatasync fallback for FS without O_DIRECT.
write_slot/read_slot convenience + write_aligned/read_into zero-copy (the QD>1 spill/reload
path gathers/scatters straight into the aligned slot buffer). verify-on-read: absent/torn/
foreign slot -> None (a MISS to recompute + reclaim), never a crash. posix_fallocate preallocate.
9/9 unit tests (real temp FS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CpL3Config (TOML/JSON): per-machine L3 disks (paths + budgets), backend, shared index
location/size, and the rank->disk mapping. disk_for_rank balances M CP ranks over N disks
(r % N; ranks share a disk gracefully — throughput is drive-count-bound) or honors an
explicit map; fail-loud on length/range mismatch. probe_disk_plp: O_DIRECT write+fdatasync
latency heuristic for the durability gate (non-PLP disk must fail loud when require_plp).
No hardcoded paths. 11/11 unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3.0 (L3 disk durable floor) foundational primitives, pure + unit-tested:
- 64B self-describing blob header (magic/version/payload-kind/content-hash/CRC),
verified on every read (verify_blob) -> torn/bit-rot slots become a MISS, not garbage.
- slot_bytes_for_page: header+payload rounded to the 4K O_DIRECT boundary.
- CpL3SlotPool: O(1) free-list over fixed-size per-payload disk page-slots (the L3
analog of CpSharedL2PageAllocator's free list), fail-loud on double-free/OOB, with
reset() for the flush_cache hook. NOT a ring buffer (eviction frees arbitrary slots).
Rank-local (a rank stores only its owned pages; L3 eviction selection rides the shared
LMDB replicated clock). 12/12 unit tests. Design: docs_internal/cp_hicache_l3_phase3_impl_design.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the interim ack_queue_len==0 band-aid with a structural fix so the perf skip is
SAFE rather than removed.
Root cause: the radix attach/defer/prune/split probes read `_node_host_write_pending`
(membership in ongoing_write_through/pending_host_backups, which are drained ASYNCHRONOUSLY
at per-rank ack completion), and `_cp_subtree_has_unprunable_state` also read the rank-local
`lock_ref`. So per-rank ack-drain timing leaked into the match_prefix truncation ->
cache_protected_len -> logical_len/len(value) -> the attach predicate. On some ranks the
prepared backup attached, on others it was dropped to the synchronous write_backup catch-up
(catch_up_all_layers draft=True), whose ack-append is per-rank. ack_write_queue then diverged
across CP ranks, so writing_check's ack_queue_len==0 early-return diverged: a strict subset
entered the writing_check_min all_reduce while the rest advanced to recv_requests' broadcast
on the same tp_cpu_group (8 ranks; dp_size==1 resets enable_dp_attention to False) -> NCCL
deadlock (prod hang: node_id=50713, all 8 GPUs 0% util).
Fix: introduce TreeNode.cp_backup_pending, a rank-replicated marker set at the rank-replicated
prepared-backup attach (_attach_prepared_cp_backup) and the guarded write_backup registration,
cleared only at the MIN-committed commit (_commit_pending_backup) / rollback. Rewire the CP
radix probes (_cp_node_split_still_pending, _split_node guard, _cp_subtree_has_unprunable_state,
_inc_hit_count) to read this marker instead of the drain-timed dicts, and DROP the lock_ref read
in the prune probe (its only cross-rank skew is the backup lock, now covered by the marker).
_node_host_write_pending stays the drain-timed accessor used only by writing_check / eviction /
the visibility check (_node_host_write_ready). With the radix-path reads replicated by
construction, len(value)==logical_len on every rank -> the prepared backup always attaches (or
rolls back) symmetrically -> the asymmetric catch-up never fires -> ack_write_queue is symmetric
-> the existing ack_queue_len==0 skip is all-or-none (no per-tick no-op MIN, best perf).
Correctness: the marker is cleared only at/after _commit_pending_backup, which runs only under
the MIN frontier, so the MIN remains the SOLE host-visibility gate (no KV-corruption class
reopened); the marker is strictly more conservative than the old dicts (stays True until commit).
Fail-fasts: double-attach raises; _split_node propagates the marker. hi_mamba unchanged (gates
on ongoing only, no async ack skip). Single-layer-draft (c3fc3ff752) preserved: the design
touches only the attach decision; the draft notifier already fires post-MoE.
Adds test_cp_hicache_symmetric_attach.py: the probes read the replicated marker not the dicts,
the prune probe ignores lock_ref, and the marker lifecycle (attach->commit->rollback) + the
double-attach guard. All pass on the fix; all fail on the pre-fix code.
Design: docs_internal/cp_hicache_symmetric_attach_design.md. Pending: ETE (GSM8K + no-hang
flood + TTFT) on the dev-cu13 container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 876a98177cd749b5a63623ba2ae8b868289b8e59)
The CP shared-KV remap caches may receive tensor views that share the same storage but represent different logical page rows. Keying only by storage pointer and shape can reuse a stale remap for another view, corrupting cache-hit materialization. Use the actual tensor data pointer plus stride, storage offset, and version so different views and mutations are not collapsed into one cache entry.
Constraint: CP shared-KV bs>1 cache-hit paths reuse small tensor views over shared backing tensors.
Rejected: Clear the remap cache on every call | would avoid aliasing but add avoidable hot-path churn and hide the identity bug.
Confidence: high
Scope-risk: narrow
Directive: Do not reduce the remap cache key back to storage pointer only; same-storage views are semantically distinct here.
Tested: g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_paged_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/speculative/test_eagle_worker_v2_cp_hidden.py
Not-tested: local pytest for CP runtime import is blocked by missing starlette in the local base environment.
(cherry picked from commit 7360ef13565dbf21428e3b121112135a2955f913)
Two latent CP shared-KV L2 correctness fixes, landed BEFORE L3 (not in an L3
commit). Surfaced by a per-CP8-instance (CP8DP2EP16) scoping review.
PREREQ-1 (CP-group scoping). The B1 commit/evict consensus collectives
(writing_check ReduceOp.MIN, placement_digest MIN/MAX + its tp_world_size<=1
entry guard, drain_storage_control_queues, the evict/prefetch MINs, the flush
barrier) AND cache_controller.prefetch_tp_group all derive from
self.tp_group/self.tp_world_size, which was params.tp_cache_group. tp_cache_group
equals the CP group ONLY at dp_size=1 (enable_dp_attention False -> tp_cpu_group,
and attn_cp_size==tp_size -> _ATTN_CP==_TP) -- the sole reason B1 works today.
Under CP8DP2 (DP attention, attn_tp_size=1) tp_cache_group is the size-1 attn-TP
group, so every `tp_world_size>1` collective silently no-ops per rank and the
placement assert self-disables -> divergent placement -> shared-slab corruption.
Fix: for CP hicache (cp_size>1) scope self.tp_group to the CP cpu group
(get_attention_cp_group().cpu_group -- already used for the slab-handle
broadcast) + self.tp_world_size to its size. A no-op handle change at dp_size=1
(same group object); the intended fix at CP8DP2. The single init-point change
propagates to every consensus collective + un-gates prefetch_tp_group + re-enables
the placement assert.
PREREQ-2 (flush reset). HiRadixCache.reset() cleared the radix tree + host pool
but never reset CpSharedL2PageAllocator -> stale free list/ranges/committed after
flush_cache (leak; the shared pool was never reclaimed). Added
CpSharedL2PageAllocator.reset() (rebuild the per-slab free list all-free, drop
ranges + committed, restore the freshly-built placement_digest) called from
cache_controller.reset() after the ack queues are cleared. Safe: flush_cache is
idle-gated (no in-flight backup/reserve). This is also L3's clear hookpoint.
Validation: new test_reset_restores_freshly_constructed_all_free_state + 89/89
pool suite (torch-2.11 container) + import smoke. PREREQ-1 is a no-op at dp_size=1
(live no-regression confirmed on the next prefill restart); CP8DP2 correctness is
by construction (CP-group membership verified) pending a 2-machine run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tier-2-GC follow-on (design §9.12 flagged vestige). The metadata's `committed`
bool and `committed_payload_layers` dict were write-only residue of the option-A
quorum the GC removed: set at construction, validated in __post_init__, copied in
split() -- but their only content/True writer was the deleted
_commit_cp_shared_l2_layers_batched. Verified tree-wide there is NO live reader
(`committed` appears only in split's `committed=self.committed` copy;
`committed_payload_layers` only in __post_init__ validation + split copy +
construction). Commit consensus under B1 is the allocator's `_committed_objects`
(MIN frontier), so `metadata.committed` was a redundant second source of truth.
CORRECTION to the original flag: `required_payloads` is NOT a vestige -- it is
LIVE (read by cache_controller.load_cp_shared_l2, the evict path hiradix:2748,
and the readback payload planning). KEPT.
Removed: the 2 dataclass fields + their __post_init__ validation + the two split()
child-copies + the cache_controller reserve-construction kwargs + 4 test
constructions/assertions. No behavior change (fields were never read). Pool suite
88/88 (syh-dev-new/torch-2.11 env) + import smoke confirm the metadata dataclass is
now {logical_len, padded_len, page_size, object_ranges, required_payloads, object_key}.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the 2b.1b shared-L2 capacity-miss SKIP with a deterministic
collective-free evict-to-fit, and deletes the #5 deadlock. This is the gate that
makes --enable-cp-shared-physical-l2-hicache safe in a real run.
evict-to-fit (_reserve_write_cp_shared_l2_evict_to_fit, hiradix): on a shared-L2
reserve capacity miss -- which raises identically on every CP rank over the
replicated free list -- snapshot the plannable shared-L2 host leaves (excluding
the node being backed up) in the replicated SLRU order (_cp_host_evict_key =
Phase-1 logical clock + node.id total-order tiebreak), release the coldest one at
a time (each a replicated free-list mutation via the proven
_evict_cp_host_for_write_admission teardown -> evict_cp_host -> allocator.release)
and retry the pooled reserve after each, stopping at the first fit.
Fragmentation-correct (free coalesces in _return_range). Identical victim
set + order + deterministic reserve => every rank evicts the same victims and
stops at the same point (design Thm 1), so NO collective is needed. The snapshot
is one-level by design (a parent promoted to a leaf mid-loop is not re-pushed;
deep cascades skip-and-retry next tick -- missed identically on all ranks).
Exhaustion -> loud rate-limited skip (transient: pinned/in-flight objects hold
the pool), never a hang.
#5 deadlock DELETED: stripped the synchronize_across_ranks all_reduce machinery
(the `while len(heap) and not all_ranks_done()` ReduceOp.MIN host_evict_done_min
closure + the param) from _evict_host_for_physical_slots -- never reached (no
caller passed True) and the headline CP-deadlock shape. evict_host(num_tokens)
still works (single-arg, equivalent non-sync loop).
Opus adversarial review = SHIP (all 7 findings PASS: determinism [_cp_host_evict_key
purely replicated, pin_expiry dormant under CP], teardown byte-equivalence + no
double-free, no infinite recursion + clean abort-on-partial, snapshot safety [no
ancestor/descendant among host leaves], bounded termination, clean #5 strip,
flag-off path untouched). Tests: new test_eight_ranks_evict_to_fit_stays_identical
(fill->miss->release-in-order->retry -> identical placement, exercises coalescing)
+ 88/88 pool suite (syh-dev-new) + import smoke.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2b.1b adopt-then-strip left the entire option-A commit-quorum apparatus
orphaned: B1's writing_check ReduceOp.MIN frontier (mark_object_committed) is
the all-ranks-done consensus, so the per-(rank,layer,payload) gather quorum it
replaced is information-redundant (design sec 2.4 Corollary). This GC removes
the apparatus and collapses the CpSharedL2PageAllocator commit model to exactly
{_ranges_by_object, _committed_objects, _free_by_payload} -- precisely what
placement_digest() hashes -- so the cross-rank digest assert now covers the
whole model and the dead quorum is structurally unrepresentable.
Removed (all proven dead under B1, confirmed by a full tree caller-sweep +
opus adversarial review = SHIP):
- allocator: commit_layer, _has_full_commit, _expected_layers_for_object_payload,
adopt_reserved_range, set_object_required_payloads; fields _commits_by_object,
_expected_ranks, _adopted_ranges, _required_payloads_by_object,
_expected_layers_by_object, _required_payloads, _expected_layers; ctor args
expected_ranks/expected_layers/required_payloads; module fns
broadcast_cp_shared_l2_decision, gather_cp_shared_l2_{preflight,commits,commit}.
reserve/split_committed_object/_drop_object simplified to ranges+committed-bit.
- cache_controller: the 3 dead _commit_cp_shared_l2_* fns, 3 imports, all 5
option-A ctor params (cp_shared_l2_{cpu_group,source_rank,broadcast_fn,
preflight_fn,commit_fn}), the per-object commit-contract builder/setter/call.
- hiradix: allocator construction updated (3 args dropped); live _split_node
caller and _get_cp_shared_l2_rank_and_group@167 intact.
- tests: split tests migrated to mark_object_committed; 4 dead-symbol tests removed.
Lost __init__ payload-has-slab validation is covered by reserve()'s own
ValueError (fail-loud at first write). Default (flag-off) path unchanged: the
allocator is never constructed when enable_cp_shared_physical_l2_hicache=False.
Net -580 LOC. Validated: 87/87 pool suite (syh-dev-new, torch) + 31/31 core
classes locally + edited-module import smoke. Out of scope (flagged separately):
the CpSharedL2NodeMetadata.{required_payloads,committed_payload_layers} vestige.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The B1 write-through: every CP rank runs the IDENTICAL deterministic CpSharedL2PageAllocator.reserve over the replicated event stream -> identical placement with NO broadcast/gather; commit rides the writing_check ReduceOp.MIN frontier (mark_object_committed) instead of a per-(rank,layer,payload) gather quorum.
cache_controller.py: _reserve_write_cp_shared_l2 stripped to every-rank reserve (dropped preflight#2/rank0-gate/broadcast#3/adopt/missing-payloads; per-object contract kept MF3; idempotent abort-on-partial SF3); _submit_write_cp_layer_states drops the 3 gather-commit calls (per-layer D2H + all-payload done-gate MF1 + per-node ack kept); submit_write_cp_all_layer drops both fallback gathers (zero-owned keeps its ack MF2).
hiradix_cache.py: 3-way merge of l2_pooling shared-L2 init (clean, 0-conflict; Phase-1 clock + E1 + reserve-budget preserved) builds the allocator/slab when the flag is on and passes it to HiCacheController WITHOUT collective fns. The #5 deadlock removed: shared_l2_capacity reserve-miss now SKIPS the backup (reactive, rank-uniform) instead of _evict_host_for_physical_slots(synchronize_across_ranks=True) -- the deterministic shared-pool evict is 2b.2. mark_object_committed wired at _commit_pending_backup (the MIN commit). New _cp_assert_placement_replicated (SGLANG_CP_HICACHE_PLACEMENT_ASSERT) in check_hicache_events: MIN/MAX-reduce placement_digest across the CP group, fail-loud on divergence (Theorem 1 runtime gate; rank-uniform entry, cannot deadlock).
environ.py: SGLANG_CP_HICACHE_PLACEMENT_ASSERT (EnvBool, default off).
Tests: 8-rank reserve-determinism (identical placement across 8 ranks + divergence-detectable) + full pool suite 91/91 + import smoke on g0033 syh-dev-new. TWO opus adversarial reviews both SHIP (cache_controller strip + hiradix wiring).
NOTES: default-off (--enable-cp-shared-physical-l2-hicache); validated for write_through policy (prod default). Capacity-miss currently SKIPS (B1 shared-pool evict = 2b.2) so the flag must stay off in real runs until 2b.2. Dead gather island (3 fns + 3 imports, opus-confirmed inert) pending GC follow-up. Design+proof: docs_internal/cp_hicache_2b_pooled_l2_b1_design.md sec 2.4/9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CpSharedL2PageAllocator.mark_object_committed(object_key) -- the B1 commit path that marks an object committed directly, bypassing the per-(rank,layer,payload) commit_layer/_has_full_commit quorum. Under B1 the all-ranks-done consensus is the writing_check ReduceOp.MIN frontier (the same barrier that releases the node write lock), so every rank calls this with the same object_key at the MIN commit point and the committed set transitions rank-uniformly (feeds placement_digest).
Resolves opus-review MF4: split_committed_object stays coherent for a B1-committed object (no per-layer commit facts) -- it adds children to _committed_objects and the empty _commits_by_object is harmless because B1 never calls _has_full_commit.
7 unit tests (pure-Python): commit-without-quorum, idempotent, unknown-raises, stat bump, digest reflects committed set + lockstep equality, split-of-marked-object coherent, release-after-mark frees pages. Full suite 88/88 green on g0033.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CpSharedL2PageAllocator.placement_digest() -- a deterministic, order-independent SHA-256 of the full replicated allocator state (free list + per-object ranges + committed set). This is the B1 proof-obligation gate from the 2b design (sec 2.4, Theorem 1): the write-through (2b.1) cross-rank assert hashes this each tick and EQ-checks it across CP ranks, turning placement-determinism into a fail-loud runtime invariant -- any reserve/release fired on a subset of ranks or with a per-rank arg changes the digest.
6 unit tests (pure-Python, no torch): determinism, changes-on-reserve, identical-op-sequence => identical-digest, divergent-op => different-digest, reserve+release restores digest (free-list coalescing), placement-order sensitivity. Full suite 81/81 green on g0033 syh-dev-new.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopt the ownerless shared-physical-L2 mechanics from origin/l2_pooling onto the Phase-1 branch, behind --enable-cp-shared-physical-l2-hicache (default off -> behavior-neutral). Foundation for the B1 collective-free allocator (2b.1/2b.2) and the L3 durable floor (Phase 3); no coordination yet.
- cp_shared_l2_pool.py (new): CpSharedL2PageAllocator (deterministic first-fit, capacity charged once, no x cp_size), hugetlbfs-2M slab primitives, position-indexed object ranges, commit-quorum data model.
- memory_pool_host.py: SharedHostTensor* slab orchestration (3-way merged; fix-output's net change to this file is blank-line-only -> additions-only, no content lost).
- server_args.py: flags + _handle_cp_shared_physical_l2_hicache_validation (3-way merged clean with Phase-1 assert + fix-output EnvField).
- test_cp_shared_l2_pool.py: 75 unit tests (allocator/slab/quorum), green on g0033 syh-dev-new; envs stub provides SGLANG_REQ_WAITING_TIMEOUT (read by Phase-1 validation).
Design + B1 proof: docs_internal/cp_hicache_2b_pooled_l2_b1_design.md (sec 2.4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make CP shared-KV HiCache eviction collective-free and scan-resistant by
removing every per-rank wall-clock input from the eviction/admission decisions
(any such input desyncs the must-be-replicated victim set / batch across the 8
CP ranks and can deadlock the collective-coupled writeback/load-back).
- Replicated logical clock: TreeNode.next_access_time() (a process-global
monotonic counter, mirrors mamba/swa's get_last_access_time) replaces
time.monotonic() as the source of last_access_time at every radix bump site
(radix_cache __init__/match/insert; hiradix CP match/insert/_insert_helper_host;
reset() zeroes it). creation_time/pin_expiry intentionally stay wall-clock.
Because match/insert events are replicated (reqs broadcast from rank 0 over the
replicated tree), last_access_time is now identical on every rank. Unique ints
also give a strict total order (no LRU tie ambiguity). No duration arithmetic
reads last_access_time, so non-CP LRU is unchanged; mamba/swa use their own
TreeNode and are untouched.
- CpReplicatedSLRUStrategy = (is_protected[hit>=2], last_access_time, node.id):
scan-resistant (cold one-shots stay probationary, evicted before reused/
protected prefixes), recency-within-segment ages out cold (not LFU), node.id
is the deterministic total-order tiebreak (heaps never compare TreeNodes).
Overridden for CP so it reaches every get_priority eviction site; the host
write-admission key _cp_host_evict_key is switched from FIFO-by-id to it.
- Co-fixes for the same per-rank-wall-clock class: pin_prefix is forbidden under
CP (its pin_expiry victim-eligibility check is per-rank wall-clock; SLRU
scan-resistance covers the benefit); the affinity head_age_s admission input is
disabled (=0.0, relying on the replicated head_defer_count bound); and a
server_args fail-fast guards CP shared-KV against SGLANG_REQ_WAITING_TIMEOUT>0
(its waiting-queue pruning is per-rank wall-clock).
Tests: test_evict_policy.py adds TestCpReplicatedSLRUStrategy (scan-resistance,
recency, total-order, full ordering) -- 30 passed in the dev-cu13 container;
test_radix_cache_unit.py adds logical-clock determinism/total-order tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A long cold-request workload flooded the prefill log (~177MB / 32K lines in
8min, all 8 CP ranks) with cp_host_reservation_plan_insufficient +
prepare_write_backup_reservation_failed even though the host pool was <42%
used and the writes physically fit.
Root cause: _free_room_deficit folds the proactive 20% free-room target
(hicache_host_free_room_ratio) into the admission deficit once a lane dips
below the 10% trigger. _evict_cp_host_for_write_admission then failed -- and
evicted nothing -- whenever it could not reclaim that full 20% target, which
is impossible while the residual is locked/pending. The lane never drained,
so every subsequent backup re-issued the same impossible demand and re-logged
on every tick. The reservation failure is itself a graceful skip, so there is
no crash; the symptom is the log storm.
Gate write-backup admission on the HARD deficit -- max(0, required-available)
over the target and draft lanes (draft_available is 2**62 with no draft pool,
a no-op) -- and evict toward the watermark best-effort, admitting whenever the
write itself fits. This drains the stuck lane and backs the request up instead
of skipping+flooding. The proactive watermark stays as the eviction target
(deficit_by_owner unchanged); it is no longer a hard admission gate. The
decision is computed from rank-replicated state on the collective-free reserve
path, so every CP rank makes the same admit/skip choice (opus-reviewed
rank-safe).
Also rate-limit the four host-reservation fallback warnings (once per 10s per
fallback_name, with a suppressed count) so any genuine exhaustion degrades
quietly instead of producing a log storm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under overlap scheduling a chunked request's final-chunk write-backup prepare
read a stale `is_chunked` (>0): that per-tick counter is decremented in
process_batch_result, which the overlap loop runs AFTER the run_batch prepare
hook. So prepare floored `backup_end` to a page boundary (the intermediate-chunk
rule) and dropped the now-complete final tail page, while the final non-chunked
insert builds the radix node at full unaligned length. The exact-equality attach
predicate (prepared.logical_len == len(value)) then failed by
(num_tokens-1) mod page_size, dropped the per-layer overlap backup, and forced
the serial all-layer catch-up (~89% of large chunked requests in prod).
Decouple the floor decision from the stale counter: the scheduler marks
`req.cp_backup_is_intermediate_chunk = (req is self.chunked_req)` in
`_prepare_hicache_write_backups_before_forward` (the live chunked_req identity
is the authoritative "will be chunked further" signal, set this tick before
run_batch), and the prepare candidate builder floors on that flag instead of
`is_chunked`. Intermediate chunks still floor; only the misclassified final
chunk now backs up its full tail and attaches the overlap backup.
Tests: update the chunked prepare test to the new flag; add a regression test
that a final chunk with stale is_chunked reserves the full tail; add an offline
repro that drives the real prepare/insert/probe and sweeps unaligned tails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A bs=1 cache-hit prefill produced deterministically wrong output (e.g. "0.5,0.5,0.5").
Root cause: regression a24111a5f4 changed the bs=1 cache-hit call sites to pass
prefix_slot_spans=[] (get_or_build_batch_slot_spans want_prefix=False) together with
prefix_pages>0. In materialize_prefix_and_reuse_current_kv_page_slots and its index twin,
the guard `if prefix_slot_spans is not None:` let the empty list shadow the prefix_pages
fallback -> prefix_spans=[] -> the cached prefix dense slots were never materialized
(both IPC and local paths gate on `if prefix_spans:`) -> attention read zero KV over the
entire reused prefix. The indexer compose dropped its prefix the same way. bs=1 only;
bs>1 (non-empty per-request spans) and fresh prefill (no prefix) were unaffected.
Fix: make the canonical per-request slot-span list the sole description of the
prefix/current regions and fail loud on a missing list. Remove all four
span-reconstruction fallbacks: prefix_slot_span (singular, dead), prefix_pages->span
(the bug), current_slot_spans=None->span (dead), and the prefetcher's copy. Both twins
now require prefix_slot_spans + current_slot_spans (raise on None; [] = genuine
no-prefix). All call sites build canonical spans via want_prefix=True (cached per-forward,
so a24111a5f4's per-layer-CPU goal is preserved by the cache, not by skipping the build);
the MLA current-only path also uses want_prefix=True to keep want_prefix uniform and the
span cache thrash-free.
Tests: migrate unit tests off the removed prefix_pages param (drop it where spans were
already passed; supply canonical spans otherwise); fix two source-string/captured-kwarg
assertions; add test_materialize_prefix_requires_explicit_prefix_slot_spans (None->raise,
explicit spans -> non-zero prefix). test_cp_shared_kv_runtime.py: 158 passed on the
g0033 container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deferred non-chunked inserts can happen while a CP HiCache per-layer backup is still in flight. Rolling back the unattached prepared backup at unfinished-request time drops the only reservation that final insertion can attach, so cache_finished_req has to start a post-forward backup and catch up all layers synchronously.\n\nKeep the prepared backup for non-chunked deferred inserts and continue rolling it back for chunked middle inserts, where the suffix may be extended by later chunks and the old backup would cover the wrong range. Document the failure mode so future changes do not rediscover the same fallback path.\n\nConstraint: CP HiCache radix split cannot mutate in-flight backup nodes.\nConstraint: Chunked prefill middle inserts still need rollback because their backup range is not final.\nRejected: Always rollback unattached backups | causes post-forward catch_up_all_layers for non-chunked deferred inserts.\nRejected: Always preserve unattached backups | can attach stale backup ranges for chunked middle inserts.\nConfidence: high\nScope-risk: narrow\nDirective: Do not clear req.cp_hicache_prepared_backup on non-chunked deferred insert without proving final insert no longer needs it.\nTested: remote cjy-glm5-new py_compile radix_cache.py\nTested: remote cjy-glm5-new pytest test_cp_hicache_metadata.py::{nonchunked preserve,chunked rollback} => 2 passed\nNot-tested: live ETE replay after restarting prefill with this exact commit
The syh rebase kept callers that expect reusable batch slot spans, while the restored CUDA IPC runtime lacked the helper and indexer still passed a symm-only writer-rank argument. Restore the batch-scoped span cache and remove the stale symm argument so the production compose path stays on CUDA IPC with exact per-request spans.\n\nConstraint: Production main-stream compose should use CUDA IPC, not symm writer-rank routing.\nRejected: Re-enable symm writer-rank parameters | benchmark showed no main-stream win and callers fail against IPC runtime contracts.\nConfidence: high\nScope-risk: narrow\nDirective: Keep slot-span metadata batch-scoped; do not rebuild Python span descriptors per layer without benchmarking.\nTested: Local py_compile for cp_shared_kv_runtime.py, cp_shared_kv_prefetch.py, nsa_indexer.py, nsa_backend.py.\nTested: Remote cjy-glm5-new pytest test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 157 passed, 2 subtests passed.\nNot-tested: Full ETE prefill/decode runtime after restarting services.
Expected no-prefetch paths were polluting production logs: no cache prefix, tiny/first-layer windows, and FP8 RAGGED top-k were being reported as fallback warnings. The prefetch contract now treats zero-prefix and first-layer misses as normal skips, while preserving warnings for non-zero misaligned prefixes and real consume misses after the first layer. The same change keeps RAGGED cache-hit prefetch eligible and records the CE/IPM prefetch contract in the plan doc.
Constraint: FP8 sparse prefill uses RAGGED top-k, but CP shared-KV prefix materialization is still page-slot based
Constraint: Layer 0 has no previous attention-window hook that can have prefetched the layer
Rejected: Warn whenever a prefetcher is absent | no-cache and too-short requests are expected synchronous paths and make logs unusable
Confidence: high
Scope-risk: moderate
Directive: Keep CP_SHARED_KV_FALLBACK warnings for unexpected contract failures only; use debug logs for expected skip paths
Tested: Local py_compile for cp_shared_kv_prefetch.py, nsa_indexer.py, nsa_backend.py
Tested: Remote cjy-glm5-new targeted regression: 3 passed, 21 warnings
Tested: Remote cjy-glm5-new full test_cp_shared_kv_runtime.py: 156 passed, 21 warnings, 2 subtests passed
Not-tested: New ETE run after prefill restart to confirm log volume reduction in production traffic
(cherry picked from commit e08e321e5929fdbb30102ec0b19c6ff0ecac7e7e)
CP shared-KV slot remaps already have forward-batch lifetime, but the IPC materialize path rebuilt owner/source/dense descriptor tensors on every layer. Cache prefix/current IPC descriptors on the token and paged slot-remap objects, keyed by layout, spans, device, descriptor kind, and prefix capacity, so all model layers can reuse the same request/batch-plan descriptors.
Constraint: Small-extend cache-hit workloads showed descriptor setup could exceed the all-reduce baseline before any IPC kernel work ran.
Rejected: Global descriptor cache | slot-remap lifetime is safer and avoids stale entries across request/batch-plan changes.
Rejected: Cache without physical page capacity | prefix descriptors encode capacity-invalid pages and must miss when capacity changes.
Confidence: high
Scope-risk: moderate
Directive: Do not reuse descriptors across different slot_logical_pages identity, CP layout, spans, device, or prefix capacity; stale descriptors can alias dense slots across requests.
Tested: Local py_compile; local git diff --check; remote g0034 cjy-glm5-new targeted descriptor tests 2 passed; remote full test_cp_shared_kv_runtime.py 146 passed, 21 warnings, 2 subtests passed.
Not-tested: Full ETE throughput/accuracy after descriptor cache; CUDA service benchmark still needed to quantify speedup.
(cherry picked from commit addd1ca1571e41458315d15304a0e841682fe8fa)
Cache-hit bs>1 current reuse can create very large dense attention buffers
while touching only a small set of current pages. The previous SGLang runtime
asked tai-kernel for a staging buffer sized like the full dense tensor, which
caused CUDA OOM before the current IPC fast path could run.
Switch token and index current IPC helpers to descriptor-compact staging: publish
the dense destination pages into compact staging slots and materialize peers
from compact source page ids back to the original dense destination pages.
Document the failure mode and the compact-staging contract so the dense-sized
contract is not reintroduced.
Constraint: CUDA + SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1 must fail fast instead of silently falling back to current-slot all_reduce
Rejected: Let staging allocation failure fall back to all_reduce | hides the bug and restores the expensive collective path
Rejected: Size staging by the full dense tensor | reproduces the 965MB staging OOM on long-prefix cache-hit batches
Confidence: high
Scope-risk: moderate
Directive: Current IPC helper source ids are compact staging ids; destination ids remain dense slot pages
Tested: Remote cjy-glm5-new PYTHONPATH=python:/mnt/beegfs/cjy/tai-kernel/python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 144 passed, 2 subtests passed
Tested: Local py_compile cp_shared_kv_runtime.py
Not-tested: Full ETE service restart with production traffic after this commit
(cherry picked from commit 906ecbe5d4f08b73242e98e2b628e26516d5b04a)
CP shared-KV cache-hit batches should compose long prefix pages and short current pages through page-slot IPC instead of falling back to dense all_reduce. Wire the runtime and prefetch consume paths to the TAI current-staging helpers, fail fast when the configured CUDA fast path cannot run, and document the bs>1 cache-hit benchmark evidence.
Constraint: bs>1 prefill must preserve the page-slot contract across fp8/bf16 and zero-lane current tails.
Rejected: Silent all_reduce fallback | hides correctness and performance regressions in production.
Confidence: medium
Scope-risk: moderate
Directive: Any future fallback in CP shared-KV CUDA fast paths must be explicit warning/fail-fast and covered by runtime tests.
Tested: Local py_compile cp_shared_kv_runtime.py and cp_shared_kv_prefetch.py; remote PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py (144 passed, 21 warnings, 2 subtests passed); remote TAI IPC benchmark fp8 bs>1 cache-hit matrix recorded in docs.
Not-tested: Full ETE mixed replay after replacing all current collectives with IPC.
(cherry picked from commit 8aa3b4ce59e0ebef5da5b0d07499a5f1d9785997)
CP shared-KV bs>1 compose must not silently fall back to dense full-buffer collectives when CUDA TAI materialize is expected. The fallback masks both correctness contract drift and severe synchronization/communication regressions, especially while comparing the symm path with the older IPC path.\n\nThis keeps CPU/unit-test fallback available, but makes production CUDA+TAI runs raise an explicit compose_v2 fail-fast for token-KV and index dense fallback. It also records the symm-vs-IPC comparison contract so barrier and collective counts are evaluated alongside elapsed time.\n\nConstraint: Production cache-hit-heavy bs>1 paths must expose unexpected dense collectives instead of silently taking them.\nRejected: Cherry-pick the old IPC branch wholesale | it conflicts with the symm compose design and would mix two transport protocols before benchmarking.\nRejected: Allow dense fallback with warning only | warning can be missed and still corrupts performance conclusions.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable dense full fallback in CUDA+TAI compose paths without a benchmark proving it is intentional and a correctness test covering cache-hit bs>1.\nTested: python -m py_compile for cp_shared_kv_runtime.py and test_cp_shared_kv_runtime.py; git diff --check.\nNot-tested: Remote container pytest/ETE; local pytest is not reliable in this workspace because dependencies such as orjson are missing.
The bs=1 MLA/index prefetchers replaced their consume-side trailing-range
NCCL all-reduce with the staging exchange: fill current rows straight into
this round's staging span (token-KV collapses to one cached index_copy;
the index fill kernel is just pointed at the staging page inverse),
cp_symm_barrier, then gather ALL current pages — this rank's own included
— from the stagings into the prefetched dense buffer. The symm+prefetcher
FAIL_FAST is gone.
Rank-uniformity moves with it: staging registration now also happens in
maybe_create (batch-logical gates, before any per-rank miss can diverge),
because with a prefetcher active the sync compose runs only on per-rank
misses and its lazy collective registration would hang. A hit/miss
divergence itself stays barrier-safe — both the prefetch consume and the
sync-compose fallback execute exactly one begin_round + barrier per
(layer, kind), and the counting barrier is shape-free (unlike the AR pair
it replaces, which would shape-mismatch).
Found by the new index test phase: the fill/remap kernel family skips
page id 0 as the SGLang dummy page, so a 0-based first staging slot was
never written. The staging layout now reserves row 0 (slot of current
page i = i + 1) for every kind, matching the convention instead of
depending on per-kernel behavior.
Launch-path cost: per-(kind,parity) peer pointer tables and the
[pool|staging] concatenations are precomputed/cached (identity pinned by
holding the pool-table reference); all prefetch descriptors, staging row
indices, and mixed_locs are built once per batch.
Validated on g0033 8xH200: 151 unit tests; 8-rank byte-exactness for
token sync symm (8 layers), index sync symm (4 layers, new phase), and
MLA + index prefetch consume_prefix_with_current vs the legacy sync
compose.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The publish-variant staging exchange measured ~equal to the default
compact-current AR (90.4 vs 86.5 ms/batch on the traced scenario): the
publish copy and the barrier serialized behind the 0.65 ms prefix
gather ate the transport win the isolated current exchange showed
(0.196 vs 0.354 ms). Fix the structure instead of the copy: current
rows are now written straight INTO the staging — the fill kernels take
their write destinations solely from page_inverse, so a per-batch
staging-remapped page inverse on the plan retargets them with zero
kernel changes — then cp_symm_barrier, then ONE slot-dense gather
covers prefix pages (pool pointers) and ALL current pages (staging
pointers, including this rank's own) through a concatenated 2*cp
pointer table where current slots carry owner = cp_size + writer and
src = staging slot. No publish copy, no prefix pre-gather, no second
gather.
The fused fill's loc outputs are dense-geometry-bound, so the token-KV
path computes mixed_locs/staging row indices once per batch (they are
layer-invariant) and the per-layer fill collapses to a single
index_copy_ into the zeroed staging span.
Benchmark (g0033 8xH200, byte-exact, idle-checked): 62.8 ms/batch vs
84.4 default Step A (-26%) and 60.8 ideal; publish variant was 88.0.
151 unit tests; 8-rank GPU byte-exactness vs v2 across 8 layers,
arena on and off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Peers only ever read the CURRENT pages of a rank's compose output — the
prefix comes straight from the IPC-registered KV pool — so the symm
region does not need to hold the whole dense buffer (pool-bound ~2.5 GB
double-buffered slab). It now holds one round of current pages in
merged-span order (extend-cap-bound, ~58-100 MB), and dense buffers
become purely rank-local (plain allocations or the optional local arena;
COMPOSE_SYMM no longer requires COMPOSE_ARENA).
Exchange per compose call: publish my written current pages
dense[page] -> staging[slot i] (slot = the page's batch current index,
identical on every rank, so peers address each other's staging with no
per-batch handshake), cp_symm_barrier, gather peers'
staging[writer][slot] -> dense[page] via the existing src!=dst page
gather. Reuse safety keeps the parity-half distance-2 argument, now on
the staging. Capacity sizing comes from the admission caps
(max_total_extend_tokens / max_batch_requests) with a pool-derived
fallback and the SYMM_HEAP_MB override; overflow fails fast
(batch-logical, hence rank-uniform).
Idea credit: laoyao0822's touched-pages-proportional staging
(906ecbe5d4), rebound onto our barrier-gated, group-agreed transport.
Validated on g0033 8xH200: 151 unit tests; 8-rank GPU byte-exactness
vs compose_v2 across 8 layers (arena on and off, parity halves
exercised); benchmark path e (real protocol) byte-exact, current-page
exchange 0.196 ms vs 0.354 ms compact-AR isolated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Radix insert can reach the pending-backup/stale-tail split probe on only a subset of CP ranks. The previous opportunistic ack drain called writing_check(), which may issue a scalar write-visibility collective while other ranks are polling inflight transfer state with a different tensor shape. That ordering mismatch matches the observed Gloo 4-vs-2 abort after CP_HICACHE_FALLBACK insert_deferred_pending_backup_split logs.
This keeps the split probe local and conservative: it leaves ready write acks queued and lets the globally ordered scheduler/cache-event path drain them. The regression test locks the contract by making any collective from the pending-split probe fail.
Constraint: CP radix insert/pending-split probing is not globally ordered across ranks.
Rejected: Drain ready write acks from the split probe | can interleave with scheduler poll collectives and corrupt distributed collective ordering.
Confidence: high
Scope-risk: narrow
Directive: Do not call writing_check() or any CP/TP collective from radix pending-split probes; drain write acks only from globally ordered paths.
Tested: Remote cjy container: PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 126 passed, 21 warnings.
Tested: Remote cjy container: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py => PY_COMPILE_OK.
Not-tested: Full ETE prefill/decode replay after this change.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Addresses the opus review of the Step A/B compose stack:
- FAIL FAST on symm + prefetcher coexistence: the MLA/index prefetchers
issue their own per-span collectives and bypass the symm exchange, so
letting them coexist would make COMPOSE_SYMM a silent no-op on every
prefetch hit (measured "parity" that never ran). maybe_build_current_
page_writer_ranks now raises when a prefetcher is attached or
SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH is set.
- Arena parity now derives from a monotonic round epoch instead of raw
layer_id: EAGLE draft layers reuse decoder layer ids, so id-parity would
stop alternating halves (draft L0 -> next forward target L0 lands on the
same half) and repeated-id rounds would keep appending into one half.
begin_round(layer_id, kind) starts a new round (other half, offsets
reset) when the id changes OR a kind repeats; regression test included.
- Per-batch writer-ranks cache on the forward batch and a presence flag in
the plan key: previously the ~bs x current-pages writer list was rebuilt
AND tuple-hashed on every layer per call site, pure launch-path waste.
- Rank-uniformity invariant documented at _symm_exchange_current_pages
(any per-rank gate must go through a group agreement first) and an
actionable arena-overflow message naming SGLANG_CP_SHARED_KV_SYMM_HEAP_MB.
Deferred follow-ups (documented in the design doc): factor the two near-
clone v2 helpers, single barrier per F-layer (~308us/batch), persistent
device-side peer-ptr tables for the gather.
182 targeted tests + 8-rank GPU byte-exactness (v2 and symm) re-validated
on g0034 syh-dev-new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the last remaining NCCL collective in the bs>1 compose layer loop
(the compact-current all-reduce) with a DeepEP-style counting barrier plus
a CUDA-IPC gather from peers' symmetric dense buffers, behind
SGLANG_CP_SHARED_KV_COMPOSE_SYMM (+ COMPOSE_ARENA, both default off).
- CpComposeArena.register_symm: fixes capacity at the pool-derived bound
(logical pages x dense page unit, overridable via
SGLANG_CP_SHARED_KV_SYMM_HEAP_MB), allocates slab + flags in CUDA-IPC
memory, exchanges handles once over the CP group; deterministic bump
carve means peer_base + my_offset addresses any peer's dense buffer with
no per-layer handshakes. Registration happens only in the token-KV
compose (uniform first-use point); growth after registration raises.
- Current pages are single-writer at page granularity under the
page-aligned in-seq split, so the exchange is the existing
gather_cuda_ipc_peer_pages with src==dst page ids and writer (compute
owner) ranks; writers are built per batch by
build_batch_current_page_writer_ranks and gated by
maybe_build_current_page_writer_ranks (page_aligned metadata required).
The barrier runs even with zero remote pages (counts must match).
- _agreed_tai_ipc_peer_ptrs: the per-rank IPC capability probe is now
agreed across the CP group (one-time MIN all-reduce per pool tensor) so
ranks can never split between gather and collective paths and deadlock
on mismatched NCCL shapes.
- ComposePlan cache re-anchored ON the slot_remap object (forward-batch
lifetime) instead of a module-level tensor-identity key, which could go
stale when a freed tensor's address is reused by the next batch.
Validation (g0034 syh-dev-new): tai-kernel cp_symm_barrier correctness
(200 adversarial iterations, rotating 10ms producer delays, phase-safety,
flags drained at quiescence) and perf (7.7us max-rank latency) both pass;
8-rank GPU test extends to the symm path - byte-identical to compose_v2
across 4 layers (parity halves exercised, slab registered); mem_cache
suite 464 passed with the only failures being a documented pre-existing
sys.modules stub pollution pair, reproduced identically with
SGLANG_CP_SHARED_KV_COMPOSE_V2=0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bs>1 partial-current materialize issued one sum-all-reduce per request
span per buffer per layer (48 collectives per F-layer at bs=12, 2880 per
batch, 419ms + 49ms launch gaps in the production trace), all inline on
the compute stream. The data is a partitioned gather, not a reduction:
every byte has exactly one producer.
compose_v2 (SGLANG_CP_SHARED_KV_COMPOSE_V2, default on) replaces this with:
- fast path: one tai-kernel CUDA-IPC slot-dense gather covering ALL prefix
spans (full-range descriptors, -1 sentinels zero-fill current slots and
replace the dense zero-fill) + ONE collective over the compact current
pages (uint8 byte view; exact because every byte is writer-exclusive).
- fallback (no peer IPC): local materialize of all prefix spans + ONE
whole-buffer sum-all-reduce (rows are still writer-exclusive pre-reduce).
IPC capability is decided once by the cached peer-pointer probe; after a
successful probe a failing gather raises (no per-call try/except).
cp_shared_kv_compose.py adds the per-batch ComposePlan descriptor cache
(layer-invariant, keyed on the slot_logical_pages identity) and the
CpComposeArena with tier-S carve discipline (deterministic bump, layer-
parity halves; default off) so the Step B symmetric-memory conversion is
a registration flip.
Microbenchmark (g0034 8xH200, traced 12-req batch, per batch): per-span
214ms -> fused AR 119ms -> IPC prefix + compact current 84ms; symm target
61ms. Validation: 143 unit tests incl. v2-contract twins, legacy siblings
and rank-merged simulations under both paths; mem_cache dir 432 passed;
8-rank GPU byte-exactness vs legacy with real NCCL + IPC (test/manual/
test_cp_shared_kv_compose_v2_8rank.py) passed with no fallback markers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reserve_write_cp undrained-ack refusal returned a bare
HiCacheWriteFailure, which _reserve_write_cp_indices_no_collective
interpreted as a host-capacity failure: with free host space the
retry path tripped the predicted-no-deficit RuntimeError (crashing
the scheduler in exactly the scenario the gate exists to handle
gracefully), and with a deficit it triggered pointless evictions
before refusing again.
Give HiCacheWriteFailure an explicit reason (default host_capacity
keeps all existing constructors/semantics); the wrapper returns
non-capacity failures immediately as skip-this-round, and a
wrapper-level test pins that undrained_ack reaches neither the retry
admission nor host eviction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
