c13d4556ffddc67ffa5f049e4b9321cab76b7a16
2336
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 laoyao0822
|
2de822661c |
Avoid decode-step syncs in min-new-token penalties
The old min_new_tokens penalizer updated logits through boolean-mask indexing. That indexing is data-dependent and can force synchronization on the decode hot path. Use an elementwise torch.where followed by inplace add so the operation stays tensorized and avoids the mask-index update path. Constraint: Keep the numeric behavior for active rows equivalent without multiplying by zero, which would turn -inf penalties into NaN. Rejected: Expanding the mask and assigning through logits[mask] | this is the synchronization pattern being removed. Confidence: high Scope-risk: narrow Directive: Do not reintroduce boolean-mask writes in decode-step penalty paths without profiling the synchronization behavior. Tested: RED/GREEN local pytest test/registered/unit/sampling/test_min_new_tokens_penalizer.py Tested: RED/GREEN remote pytest in cjy-glm5-new for min_new_tokens penalizer together with related regression tests Tested: git diff --check; py_compile for min_new_tokens.py Not-tested: Full decode throughput benchmark |
||
|
laoyao0822
|
132561a151 |
Preserve speculative finish correctness across reasoning and stop strings
ReasonerGrammarObject wraps an inner grammar, but disaggregated prebuilt replay checks the wrapper current_token to avoid accepting an already accepted token twice. Track the token on the wrapper itself so reasoning grammar follows the same contract as XGrammar. Speculative decode can accept a stop string and EOS in one step. Check stop strings before token-based EOS after sanitizing invalid token ids, and set finished_len at the matched stop position so trailing accepted tokens do not leak. Constraint: Current branch predates upstream helper methods for locating string stop positions, so the stop-string fix is manually ported instead of cherry-picked. Rejected: Direct cherry-pick of bbc853df46 | current schedule_batch.py lacks the upstream helper context. Confidence: high Scope-risk: moderate Directive: Keep vocab-boundary sanitization before string decoding; do not move token-based EOS ahead of stop-string checks without a same-step speculative regression test. Tested: RED/GREEN remote pytest in cjy-glm5-new for constrained current_token and stop-string speculative tests Tested: git diff --check; py_compile for reasoner_grammar_backend.py and schedule_batch.py Not-tested: Full scheduler/disaggregation integration suite |
||
|
laoyao0822
|
f3cf95c0f1 |
Release decode transfer state on abort paths
Decode prealloc and transfer queues own the receiver lifetime once a request leaves the queue. The abort and transfer-failure paths were removing requests after streaming/releasing KV state without clearing the receiver, leaving backend-specific request tracking behind. The scheduler idle check also ignored decode retracted requests, so idle housekeeping could run while decode handoff state still existed. Constraint: This is a manual port of upstream 18989f3d48 onto a locally diverged decode queue implementation. Rejected: Direct cherry-pick | current decode queue compaction and streamer APIs differ from upstream. Confidence: high Scope-risk: narrow Directive: Any path that removes a DecodeRequest from prealloc or transfer ownership must clear its kv_receiver unless ownership is explicitly transferred. Tested: remote cjy-glm5-new pytest test/registered/unit/disaggregation/test_decode_queue_compaction.py -q: 16 passed Tested: remote cjy-glm5-new pytest test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/observability/test_scheduler_metrics_load.py -q: 17 passed Tested: git diff --check on modified files Not-tested: full disaggregated decode E2E under live traffic |
||
|
laoyao0822
|
ac47eb61c9 |
Reuse draft MTP indexer topk on spec v1 EAGLE
Spec v1 EAGLE now follows the same model-config-gated MTP index reuse contract as spec v2. This lets models that declare index_share_for_mtp_iteration reuse the first draft step's NSA/DSA topk indices across internal MTP iterations while keeping the unsafe topk>1 case disabled. Constraint: select_top_k_tokens can reorder hidden rows when topk > 1, so carried topk indices are only valid under topk == 1. Rejected: Enable reuse unconditionally | models without the config flag may not have compatible MTP index semantics. Rejected: Broaden to target-to-draft index reuse | separate semantic change with different correctness risks. Confidence: high Scope-risk: narrow Directive: Keep spec v1 and spec v2 MTP index reuse semantics aligned, including the topk==1 guard and per-draft-forward cleanup. Tested: python -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q Tested: python -m py_compile python/sglang/srt/speculative/eagle_worker.py test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py Not-tested: full spec v1 GLM5 online throughput run |
||
|
laoyao0822
|
df3cc9abf8 |
Reuse draft MTP indexer topk only when model config allows
EAGLE V2 draft MTP can avoid recomputing NSA/DSA indexer topk across internal draft iterations when the model declares that those indices are shareable. The port follows the upstream narrow contract: store per-forward topk on ForwardBatch, enable reuse only for topk=1, and clear the transient state after draft_forward. Constraint: topk > 1 reorders hidden rows in select_top_k_tokens, so carried topk indices would no longer match the hidden states. Rejected: Reuse target-side topk for draft | broader semantic change not covered by the upstream fix or current tests. Rejected: Skip loading draft indexer weights | separate memory optimization with correctness risk for models that do not enable MTP index sharing. Confidence: high Scope-risk: narrow Directive: Do not enable index_share_for_mtp_iteration without preserving the topk==1 guard and per-draft-forward cleanup. Tested: python -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q Tested: python -m py_compile python/sglang/srt/speculative/eagle_worker_v2.py python/sglang/srt/models/deepseek_nextn.py python/sglang/srt/model_executor/forward_batch_info.py test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py Not-tested: full GLM5 EAGLE throughput/accuracy run |
||
|
laoyao0822
|
648a33ab30 |
Remove NSA spec-v2 graph metadata host syncs
NSA spec-v2 draft-extend graph replay was still using host-derived sequence lengths and the draft-decode backend allowlist. That kept NSA on eager draft-extend for spec-v2 and left seq_lens_cpu.max()/list-to-GPU tensor construction on the decode critical path. This ports the small upstream DSA metadata fixes into the local NSA backend: size the captured graph page table to req_to_token width, use the static captured page-table width for graph replay, split v2 draft-extend from variable-length v1 draft-extend, and decide draft-extend graph support from the prefill-style backend. Constraint: Current branch does not have the full upstream needs_cpu_seq_lens scheduler/FutureMap infra. Rejected: Cherry-pick the full DSA fused metadata generation series | too broad and overlaps with local NSA fused metadata-copy code. Confidence: medium Scope-risk: moderate Directive: Do not collapse DRAFT_EXTEND and DRAFT_EXTEND_V2 here; v1 keeps variable accept lengths while v2 must stay graph-static. Tested: local pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q (19 passed) Tested: local py_compile on nsa_backend.py, nsa_backend_mtp_precompute.py, eagle_worker_v2.py Tested: remote g0034 cjy-glm5-new pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q (19 passed) Tested: remote g0034 cjy-glm5-new py_compile on nsa_backend.py, nsa_backend_mtp_precompute.py, eagle_worker_v2.py Not-tested: full decode E2E with SGLANG_ENABLE_SPEC_V2=1 |
||
|
laoyao0822
|
4414db594c |
Reduce spec-v2 overlap copy overhead
Spec-v2 decode was paying unnecessary scheduler-side copy and replay costs: hidden states were copied back to CPU even when not requested, overlap result D2H ran on the forward stream, and draft graph replay issued several small copies separately. Port the focused upstream optimizations without taking the larger spec-v2 relay refactor: gate hidden-state D2H on return_hidden_states, run result copies on copy_stream with pinned async D2H, use stride arange for draft select_index, and group small draft graph replay copies while keeping the large hidden-state DMA copy separate. Constraint: Current branch carries local GLM/NSA/PD/CP changes, so upstream spec-v2 large refactors are not safe to cherry-pick wholesale. Rejected: Cherry-pick upstream spec-v2 relay/dataclass refactor | too broad for this performance fix and conflicts with current branch structure Rejected: Copy hidden states unconditionally | default chat output does not consume them after draft extend Confidence: medium Scope-risk: moderate Directive: Keep result-copy lifetime tied to copy_done; do not move copy_to_cpu back onto forward_stream without measuring decode throughput. Tested: local py_compile for changed production files and new test Tested: local git diff --check Tested: remote g0034 cjy-glm5-new as ubuntu on /sgl-workspace/sglang-tai: pytest -q -p no:cacheprovider test_generation_batch_result_copy.py test_eagle_worker_v2_cp_hidden.py test_spec_utils.py Not-tested: full two-node decode throughput A/B after restart |
||
|
laoyao0822andOmX
|
c6b99f6060 |
Stabilize spec-v2 draft graph metadata
Spec-v2 draft extend can receive token ids from producers whose dtype is not already int64, while DP collective paths require a stable integer dtype across ranks. EAGLE draft CUDA graph replay also pads raw batches to a captured batch size, so the metadata/replay path must see seq_lens_sum consistent with the padded seq_lens and then restore the caller-visible raw value. Constraint: Keep this as a narrow correctness port from upstream rather than pulling the larger spec-v2 refactor chain. Rejected: Cherry-pick broader attention-backend and decode-result refactors | current branch lacks the same upstream forward-context scaffolding and would require a separate port. Confidence: high Scope-risk: narrow Directive: Do not remove the seq_lens_sum restore without rechecking padded EAGLE draft CUDA graph metadata construction. Tested: python -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q Tested: remote g0034/cjy-glm5-new PYTHONPATH=python python3 -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q Not-tested: full multi-node GLM5 spec-v2 decode startup smoke Co-authored-by: OmX <omx@oh-my-codex.dev> |
||
|
laoyao0822
|
a27114d9dc |
Prevent spec-v2 decode warmup races
Port the fix-decode spec-v2 ownership and plan-stream fixes onto the current branch. Draft extend now keeps the scheduler-owned batch lengths committed until acceptance, binds the draft runner to the draft-extend attention backend, keeps speculative KV allocation monotonic, and lets non-graph target verify initialize metadata after DP padding. The worker also records rebound tensors on the forward stream and orders plan-stream metadata work after current-stream inputs are available. Constraint: fix-decode commits cd8e47ed9c and 60e3956d9c address CUDA illegal-address failures in spec-v2 decode warmup paths. Rejected: Cherry-pick the commits blindly | the current branch has intervening decode changes, so a minimal manual port kept the patch surface to the affected files. Confidence: medium Scope-risk: moderate Directive: Do not move target-verify non-graph metadata initialization back into prepare_for_v2_verify without validating DP padding and NSA metadata ordering. Tested: RED local pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q failed 8/8 before production changes. Tested: PYTHONPATH=python python3 -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q passed 8/8 locally. Tested: PYTHONPATH=python python3 -m py_compile python/sglang/srt/speculative/eagle_info_v2.py python/sglang/srt/speculative/eagle_worker_v2.py python/sglang/srt/speculative/spec_utils.py passed locally. Tested: Remote g0034:cjy-glm5-new pytest for test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py passed 8/8, plus remote py_compile for the three speculative modules. Tested: Local and remote sha256 sums matched for all four synced files. Not-tested: Full spec-v2 decode server restart/warmup under production traffic. |
||
 leaveletandClaude Opus 4.8
|
014948c5d7 |
CP shared-KV: global-position-rotated zigzag owner-lane balancing (default on)
In NSA in-seq-split prefill-CP shared-KV the owner builder restarted every
chunk/request at owner 0, starving physical lane 0 -> load-back livelock (g0057:
~64k cp_load_back_owner_lane_capacity_failed + ~4k owner_lane_exhausted). Rotate
the owner/compute assignment by phase=((extend_prefix_len//page_size)+req_index)
%cp_size so chunks/requests/draft pages spread across lanes, keeping owner==compute
(the local_loc_owner_mismatch invariant) by rotating the zigzag compute split by the
same phase. Cache-safe: reload replays recorded page_owners, never recomputes phase.
- owner builder + helpers (cp_owner_lane_phase two-term prefix+req_index,
cp_in_seq_reverse_map, cp_rotated_per_rank_actual_token) in
cp_shared_kv_compute_owner.py
- zigzag compute split, gather-back reverse map, bs>1 torch rerange, last-token
owner, request_phases on NSAContextParallelMetadata, scalar-path + last-token
fail-loud guards in nsa/utils.py
- mqa-logits prefill buffer estimator made rotation-aware (per-request phase) so it
no longer under-sizes the admission budget
- flag SGLANG_CP_SHARED_KV_OWNER_ROTATION (default True; =0 is the legacy escape
hatch -- phase forced to 0 = byte-for-byte legacy, needs no new kernel)
- startup require_tai_kernel_version("0.0.2") gate (utils/common.py, model_runner)
- unit tests (11): tiny-chunk pathology [40,0x7]->[5x8], synchronized-batch spread
[0x8]->[0..7], gather-back reassembly, per-rank-token
Requires tai-kernel >= 0.0.2 (phase-aware in_seq_all_gather_rerange). Reviewed by 3
adversarial agents (owner==compute consistency, gather-back equivalence, integration
safety): SHIP-WITH-NITS, all findings fixed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
wxiwnd
|
01d6ceff8f | fix(nsa): guard cp split to extend batches | ||
|
laoyao0822andOmX
|
c739ea667d |
Keep CP HiCache backup state replicated on radix nodes
Rank-local CP HiCache async write queues can drain at different times across ranks. Radix structural decisions that consult those local queues can diverge, so backup-pending state now lives on the replicated radix node and uses a separate backup lock ref from request locks. Commit and rollback clear the replicated marker, and split/prune/hit-count probes use the marker rather than local pending dictionaries. Constraint: CP ranks must make identical radix structural decisions even when local async write acknowledgements drain at different times Rejected: Drain write acknowledgements before every split/prune decision | still rank-local and adds hot-path synchronization Rejected: Treat backup lock_ref as a normal request lock | hides the distinction between request protection and backup-pending protection Confidence: high Scope-risk: moderate Directive: Do not derive CP HiCache split/prune/write-through decisions from pending_host_backups or ongoing_write_through without a replicated node marker Tested: local py_compile python/sglang/srt/mem_cache/hiradix_cache.py python/sglang/srt/mem_cache/radix_cache.py Tested: local git diff --check Tested: remote cjy-glm5-new pytest replicated marker test plus targeted CP HiCache metadata subset, 10 passed Not-tested: full test_cp_hicache_metadata.py because current branch has unrelated fixture drift around enable_cp_l3/cp_shared_l2_page_allocator/running-count setup Co-authored-by: OmX <omx@oh-my-codex.dev> |
||
|
leaveletandClaude Opus 4.8
|
144ae7acb1 |
Regression test: CP HiCache L3 reload same-tick admit->reserve race
Drives _cp_l3_reserve_reloads directly against a real CpSharedL2PageAllocator: - same-tick state (l3rl object committed-and-live + node attached -> attach_ok False) no longer raises 'duplicate live reservation'; the candidate is dropped and the request left un-held (re-matches the inserted node). - WITHOUT the recheck (attach_ok forced True = pre-fix path) the identical state raises the duplicate error -> proves the recheck is load-bearing. - an in-flight reload still piggybacks (recheck sits after the dedup, not before). - a genuine clean miss still reserves (no false-positive starvation). CPU unit test (register_cpu_ci, stage-a-test-cpu); runs in the dev-cu13 container. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
fdc23f7d61 |
CP HiCache: redesign owner-lane eviction planner as a lazy-re-evaluation heap (O(n log n))
Replaces the O(victims*candidates) per-iteration greedy argmin rescan in _plan_cp_load_back_owner_lane_evictions with a leaf-up min-heap. The score (-contribution, -unlock, slru_priority, node.id) depends on the current deficits, so a static heap is not equivalent; instead use LAZY RE-EVALUATION: a popped entry is consumed only if its score still matches the node's current score (deficits unchanged since push), else it is re-pushed with the fresh score. Stale scores are always optimistic (contribution = sum(min(counts[o], deficits[o])) and the ancestor-unlock contribution only shrink as deficits shrink), so the first up-to-date popped entry is exactly the global argmin the full rescan would have picked -> PROVABLY EQUIVALENT, with the same leaf-up eligibility + ancestor-unlock + parent-push. Determinism/rank-uniformity preserved (selection decided by the total-ordered score; the heap insertion seq only orders structurally-equal tuples). Equivalence proven by test: a verbatim reference greedy + a randomized property test (300 seeds, single- AND multi-owner deficits, non-uniform counts exercising the lazy-re-eval boundary, varied SLRU priorities) asserting byte-identical (victims, planned_freed, remaining), plus an explicit leaf-up + ancestor-unlock case (child-then-parent). 14 planner tests pass (1 pre-existing unrelated EAGLE-tail failure unchanged). Micro-bench (V100, candidates=2000 deficit=7877, benchmark/hicache/bench_cp_owner_lane_planner.py): A. ORIGINAL (.item() x cp, no memo) 50.06s B. memo + bincount (greedy) 0.494s (101x) C. lazy-re-eval heap 0.162s (309x; 3.0x over B) selection A==B==C identical Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
4e9b4d05c0 |
CP HiCache: kill the O(victims*candidates) GPU-sync storm in the owner-lane eviction planner
A CP4 prefill server hung 79s in _plan_cp_load_back_owner_lane_evictions (the [HiCache-load]
slow-scan) and was killed by the detokenizer health-check. Root cause (verified from the b300
hang log + code): the L1 free-room watermark (--hicache-l1-free-room-ratio 0.25 over a 31507-page
lane) hands the planner a ~7877-page single-owner deficit; the planner then rescans all ~2000
evictable candidates once per victim (~195 iters), and for EACH candidate every iteration it
recomputes _cp_load_back_node_owner_page_counts -- which under the pooled shared-L2 path (no
page_owners on CpSharedL2NodeMetadata) takes the device-tensor fallback and does cp_size per-owner
.item() device->host syncs. That is ~195 * 2000 * 4 ~= 1.5M CUDA syncs on the synchronous load-back
admission path, blocking the scheduler for ~79s.
Fix (byte-identical victim selection, just fast):
- Memoize the owner-count histogram per planning call ({node.id: counts}); the counts are invariant
while node.value is fixed (the plan does not mutate values), so node.id is a safe key for the plan's
duration. Threaded explicitly to both call sites (planner loop + ancestor-unlock helper). Turns
O(victims*candidates) recomputes into O(distinct nodes).
- Replace the cp_size per-owner sum().item() loop with one bincount().tolist() device->host sync.
Net: ~79s -> ~1s; the eviction plan (victims, planned_freed) is unchanged. bincount == the per-owner
loop proven over 8000 random vectors incl. the (-1)%cp==cp-1 zero-loc edge. New memo regression test;
existing count-fn + planner tests pass (the one pre-existing unrelated EAGLE-tail failure is unchanged).
(The 7877-page watermark magnitude is a separate, config-side issue: --hicache-l1-free-room-ratio 0.25
reserves ~25% of a ~2M-token lane -- ~30x more than a 64K chunk needs; lower it.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
leaveletandClaude Opus 4.8
|
ea4ce31f95 |
CP HiCache L3 3.4: cold-start (restart durability) — load|clear via SGLANG_CP_L3_COLD_START
On restart the LMDB index + disk-slab blobs persist, but CpL3Store.from_config built the slot
pools all-free and the GC LRU empty -> the next spill re-hands-out a slot the durable index still
references (clobbers a live blob) and GC never reclaims the carried-over entries. Neither a clean
start nor a durable reload was actually realized.
connect() now applies a cold-start policy BEFORE the bg threads start (the write thread is the sole
pool/GC owner, so the single-threaded reconcile must precede it):
- clear (default): wipe the persisted index + reset the pools/GC -> genuinely empty start (disk
blobs are inert, overwritten lazily on slot reuse).
- load: rebuild this rank's slot free-list + GC LRU from the durable disk blobs. Drive the scan
from the rank's OWN slab file (header-only reads) so it never inspects another rank's slots even
when ranks share a disk; a slot is LIVE iff its blob header parses AND the shared index still maps
that content hash back to this exact slot -> occupy + seed the GC LRU with the durable last_access;
orphan/unwritten slots stay free. Header-only (no payload CRC); reload-time verify-on-read still
fail-softs a torn payload. The L3 durable floor now survives a process restart.
Primitives: CpL3SlotPool.rebuild_from_allocated (O(num_slots) bulk-occupy) + CpL3DiskSlab.read_header
(one aligned block, not the multi-MB slot). Env SGLANG_CP_L3_COLD_START (default "clear"), read in
_maybe_init_cp_l3 and passed to connect(). Tests: 4 cold-start e2e (load rebuilds the floor + no slot
collision after a fresh spill; GC LRU rebuilt; clear starts empty; unknown mode fails loud) + 2 unit
(rebuild_from_allocated, read_header). 23 L3 store/disk/posix tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
leaveletandClaude Opus 4.8
|
d6aabace90 |
CP HiCache: fix L3-reload partial-prefix node split crash (well-form the node)
A radix split crashed: split_committed_object got split_pages=172 (from the node
KEY length) for a target_kv object of num_pages=1 (the host backup), raising
"split page count must leave at least one page on each side". Root cause: the L3
reload inserts a MALFORMED node -- _cp_l3_reserve_reloads stores the FULL n_full-
page suffix key but only the C L3-durable pages are reserved/hashed/host-backed
(C < n_full when L3 holds a partial durable prefix via page-level GC or partial
spill). The node then has len(key)//page = n_full but host_len//page =
object.num_pages = len(hash_value) = C, violating the radix invariant every
consumer assumes; the split derives split_pages from the long key and overruns the
C-page object. Pre-existing (L3 3.2,
|
||
|
leaveletandClaude Opus 4.8
|
dbc5ebbaa0 |
CP HiCache: multi-slab host KV cache to respect cudaHostRegister per-call ceiling
A single cudaHostRegister over a >~1 TB host buffer fails with cudaErrorMemoryAllocation on B300 (hard per-call ceiling: 512 GiB OK, 1024 GiB FAIL), crashing the hicache_size=1600 (~1.5 TB CP shared-L2 slab) prefill at startup. The registration cannot simply be chunked: a memcpy (cudaMemcpyBatchAsync, the CP-L2 H2D/D2H transfer) fails with cudaErrorInvalidValue when its host range straddles a registration boundary (verified empirically on b300-049). Fix: physically split the host cache into multiple page-aligned slabs, each <= a safe single-registration size (default 480 GiB, env SGLANG_CP_HICACHE_MAX_SLAB_GB), reusing the existing SharedHostTensorGroupAllocator + per-slab transfer splitting (_host_transfer_segments). Each slab is one whole registration and no transfer crosses a boundary; small configs (hicache_size<=400) stay single-slab unchanged. - memory_pool_host.py: add cp_hicache_max_single_register_bytes() + the fail-loud _check_single_cuda_host_register_size guard; revert the (transfer-unsafe) registration chunking back to one cudaHostRegister per buffer/slab. - hiradix_cache.py: _cp_shared_l2_slab_pages_by_payload auto-caps each payload's slab <= the ceiling so large caches auto-split. - cp_l3_slab_accessor.py: CpSharedL2SlabAccessor is now slab-count-aware (CpL3SlabSpan + per-slab dispatch via global_base_page; per-slab layer stride); _cp_l3_slab_spans rewires _maybe_init_cp_l3 off the single-slab assumption. L3 disk slabs / slot pool / LMDB index / GC are content-addressed and unchanged. Tests: multi-slab accessor incl. a non-circular torch.frombuffer-layout check; a real allocate_group + _cp_l3_slab_spans roundtrip; slab-cap auto-split; L3 store cross-slab spill/reload. Reviewed by 3 adversarial agents, no correctness bugs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leavelet
|
0025b0e819 | fix: count huge pages | ||
|
leaveletandClaude Opus 4.8
|
6114c3e3e5 |
HiCache host pool: check hugepages, not regular RAM, for the hugetlbfs-backed (CP shared-L2) pool
The host-pool size check used psutil.virtual_memory().available -- regular RAM -- unconditionally. But
hugepages are carved OUT of regular RAM, so reserving e.g. 1 TB of 2M hugepages drops psutil.available
by ~1 TB, and the check then spuriously fails ("Not enough host memory ... only have 897 GB free") even
though the slab is allocated from the hugepages, not regular RAM. It was inspecting the wrong pool.
Two fixes:
- memory_pool_host.py: gate the psutil check on the DEFAULT (regular-malloc) allocator. A custom
host_tensor_allocator (the CP shared-L2 slab over hugetlbfs) owns its own capacity check, so skip the
regular-RAM check for it.
- cp_shared_l2_pool.py: add cp_shared_l2_hugetlbfs_free_bytes() + check_cp_shared_l2_hugetlbfs_capacity()
(statvfs the hugetlbfs mount: f_frsize = hugepage size, f_bavail = free hugepages) and call it in
create_cp_shared_host_slab before ftruncate/mmap. Now insufficient hugepages fail LOUD with a clean,
actionable message ("insufficient hugepages in /dev/hugepages: need X GB (N x 2M pages) but only Y GB
free; reserve more or reduce --hicache-size") instead of a cryptic mmap ENOMEM. Per-slab + sequential,
so each payload's slab sees the free remaining after the prior ones.
Unit-tested (capacity check: tiny fits, over-capacity fails loud). Pre-existing 2b L2-pooling code (not L3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
leaveletandClaude Opus 4.8
|
5163ba7bb9 |
CP HiCache: expose L2 + L3 usage metrics (the mainline storage metrics are dead under CP)
Under CP, enable_storage=False, so StorageMetricsCollector (prefetch/backup tokens + bandwidth) never emits, and L2 pool usage was debug-log-only (_cp_maybe_log_lane_stats) while L3 was invisible. Add a CpHiCacheMetricsCollector covering both, gated on enable_metrics alone, per-rank via the tp_rank label. - CpL3Store.stats(): per-payload slot occupancy (used,total) + cumulative counters (spill_pages, reload_pages, gc_reclaimed, write_errors -- incremented on the bg threads) + instantaneous inflight (spill/reload) + the four bg queue depths. Counters are monotonic (NOT reset on clear()). - CpHiCacheMetricsCollector (metrics_collector.py): gauges for instantaneous state (sglang:cp_l2_used/total_pages, cp_l3_used/total_slots, cp_l3_inflight_*, cp_l3_queue_depth) + counters for cumulative work (cp_l2_objects_committed/evicted_total, cp_l3_spill/reload_pages_total, cp_l3_gc_reclaimed_total, cp_l3_write_errors_total). Counters fed via a monotonic .inc(delta) off the store's cumulative totals. - hiradix: create the collector in _maybe_init_cp_l3 (once, per-rank labels from cache_controller); push periodically (~1s, per-rank wall-clock -- metrics need no rank-uniformity) from check_hicache_events via _cp_maybe_collect_metrics, reading allocator.occupancy_by_payload()/stats() + store.stats(). Off the data path. store.stats() unit-tested (occupancy + spill/reload counters; 10 store tests green). The collector + wiring are correct-by-pattern (mirror StorageMetricsCollector) and validate on the live server (3.4; prometheus_client not in the unit env). Directly useful for 3.4: watch L2/L3 fill + GC + reload hits under cachebench. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
e96cfa6cbd |
L3 3.3: continuous-background, communication-free, page-level LRU GC (+ wire the reload touch)
Replaces §9.3's object-level rank-uniform eviction with per-rank, page-level LRU GC running ON THE
WRITE THREAD (the sole slot-pool owner -> no lock, no second owner). Genuinely communication-free:
each (hash,payload) is written/read/deleted by exactly its one owner (owner = i%cp_size, and the
prefix-chained content hash makes one-hash = one-position = one-owner), so the shared LMDB's own
consistency substitutes for collectives within L3 (design §9.5/§9.6). GC touches only L3, adds zero
collectives; the spill/reload MINs that remain are all the L2 bridge.
cp_l3_store.py:
- Per-payload lazy-deletion min-heap `_gc_heap` of (last_access, seq, hash) + authoritative
`_gc_current{hash->(last_access,slot)}`. Add on write, touch via `_touch_q`, reclaim coldest.
- The write loop, each iteration: drain touches -> _gc_collect (reclaim coldest of any pool over the
0.90 start watermark down to 0.85, bounded budget) BEFORE the next write, so spill never hits a full
pool (no evict burst on the critical path). Continuous + proactive, symmetric with the spill.
- Reclaim deletes the index entry (durable) BEFORE freeing the slot, so a racing reload reads None or a
CRC-mismatched reused slot -> miss -> recompute (fail-soft); no lock / in-flight-exclusion needed.
- submit_touch (scheduler->write thread); clear() resets the GC structures.
hiradix_cache.py: wire the reload `touch` (the 3.2 gap -- hit_count/last_access never updated): at
_cp_l3_admit_reload, bump the L3 last_access of THIS rank's owned reloaded pages (ALL pages, not just
the tail, so a prefix ages uniformly). _cp_l3_insert_reloaded_node now returns the node (its
last_access seeds the touch).
Sharing handled optimally for free (page = one entry, last_access = max over touchers; prefix pages
age together; shared-hot survives) -- no object-record table, no refcount, no content-keying. 56 L3
unit tests green (new: GC reclaims coldest-to-watermark + no slot leak; touch keeps a page warm).
Cold-rebuild (scan slab headers + LMDB) is designed (§9.6) but deferred to 3.4 (crash-recovery).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
leaveletandClaude Opus 4.8
|
864b1c808e |
L3 3.2: async disk->L2 reload + request hold (Model B), zero new collectives, EAGLE-correct
On a radix miss, reload the evicted-but-L3-durable suffix from disk into L2 + insert a fresh radix node, holding the triggering request until then (mirrors the storage-prefetch hold, which is disabled under CP). The reloaded prefix re-enters the radix as a normal L2 node, so the existing load_back serves L2->L1 -- no new device path. Design (docs_internal/cp_hicache_l3_phase3_impl_design.md §9, refined for minimal sync): - ENTRY (scheduler _prefetch_kvcache, enable_cp_l3 branch): mark a miss-with-suffix as a reload candidate (rank-uniform: requests are broadcast). cp_l3_reload_lookup computes the suffix's full-page content hashes -- the SAME SHA chain spill wrote (compute_node_hash_values). - RESERVE + ADMIT fold into the existing per-tick _drain_l3_control_queues MINs -- ZERO new collectives: the candidates' per-rank exists_prefix counts ride the qsize MIN vector -> a rank-uniform agreed count C (this reconciles the async-LMDB read-skew); reload-ack oks ride the durable ok-AND. Reserve is then collective-free (deterministic evict-to-fit mirroring _reserve_write_cp_shared_l2_evict_to_fit); the request waits non-blocking in waiting_queue (check_cp_l3_reload_progress skip) and is admitted + re-matched the SAME tick the reload lands (check_hicache_events runs before batch formation). CP-aware insert only on a still-clean attach (else abort+recompute). Capped (cp_l3_reload_max_inflight) + content-key deduped (piggyback). EAGLE/bigram (opus-studied, PROVABLY correct -- not deferred): the radix key is bigram-converted over the whole request, and convert_to_bigram_key(fill_ids[M:]) == bigrams(fill_ids)[M:] exactly (the boundary bigram belongs to the matched prefix), so the bigram-converted suffix hashes reproduce node.hash_value. Verified vs source + a new slice-identity regression test. Opus review (FIX-THEN-SHIP, no CRITICAL; the 4 crash surfaces -- MIN-vector shape, read-skew, attach-anchor, placement-digest -- traced clean) + independently re-verified; fixes folded: - M1: cp_l3_release_request clears reload state on request abort/timeout/preempt (no leak). - H1: rank-uniform per-op TTL bound releases a held request if its reload op never acks (the default SGLANG_REQ_WAITING_TIMEOUT is off); the reservation frees safely at the (late) ack. - L1: fail-soft anchor guard at insert (re-derive the first suffix hash from lhn's live hash). - empty-waiters reload aborts (frees L2) instead of inserting an unrequested node. Imports hoisted to top-level (get_hash_str -- hicache_storage is a leaf module, no cycle; convert_to_bigram_key already top-level), no inline imports. py_compile clean; 54 L3 unit tests green. Reload triggering (submit_reload/exists_prefix/free_object) now wired (was write-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
815da6c4e5 |
L3 3.1: redesign spill — 2-phase staging pipeline + proactive FIFO (fix evicted=0 starvation)
The first 3.1 (
|
||
|
leavelet
|
f353cf4702 |
Revert "Fix CP HiCache writing_check deadlock at the source: rank-replicated symmetric attach"
This reverts commit
|
||
|
leaveletandClaude Opus 4.8
|
de1d1e0af2 |
L3 3.0: CpL3Store orchestrator (async spill/reload, object-granular acks)
Ties config -> per-rank disk slabs + slot pools + shared LMDB index + slab accessors. Background spill/reload threads (off the scheduler tick, storage-template shape); object-granular acks (one per object after all owned pages durable; zero-owned ranks ack in lockstep) via ack queues + has_inflight, so the caller's CP-cpu-group MIN-drain frees the same objects on every rank. Spill = gather->O_DIRECT write->durable index (data->fdatasync->index-commit ordering) with content-hash dedup; reload = index lookup->read_into->scatter (verify-on-read). free_object frees owned slots+index (3.3 eviction consumes); clear() is the flush_cache hook (stop/drain/reset/restart). from_config splits the disk budget across ranks-on-disk x payloads (equal slot count). PLP gate at connect(). Model B: store is content-addressed + rank-local I/O; only the LMDB is shared. End-to-end test (spill/exists/evict/reload byte-exact/dedup/free/clear) 5/5 (venv lmdb). + accessor n_layers/page_num accessors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
2ea3728a43 |
L3 3.0: host-slab accessor (layer_page_first <-> contiguous blob)
CpSharedL2SlabAccessor (HIGH-1): memcpy-gather a page's N_layers strided slab slices into a contiguous buffer (spill) + scatter a contiguous blob back into the strided slices (reload) — the cheap RAM bridge that lets host stay layer_page_first (inference perf) while disk is page-contiguous. Generic CpL3SlabLayout(n_layers, page_num, slice_bytes) with verified-from-source factories: for_mla (layer_num,page_num,page_size,kv_cache_dim,itemsize -> 2.74 MiB/page) and for_index (n_active_layers,indexer_page_num,indexer_page_stride -> 0.17/0.63 MiB/page). gather_into/ scatter_from take an offset so the spill writes straight after the 64B slot header (zero extra copy). Pure (mmap+ints), no torch. 8/8 tests. Canonical layer 0..N-1 order shared by spill+reload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
99a695d747 |
L3 3.0: POSIX disk backend (CpL3DiskSlab, O_DIRECT page slots)
Preallocated disk-slab file per (payload_kind, drive); a page = one contiguous blob (64B header + gathered payload + pad) in a 4K-aligned slot. O_DIRECT single-transfer write/read (measured best vs vectored), buffered+fdatasync fallback for FS without O_DIRECT. write_slot/read_slot convenience + write_aligned/read_into zero-copy (the QD>1 spill/reload path gathers/scatters straight into the aligned slot buffer). verify-on-read: absent/torn/ foreign slot -> None (a MISS to recompute + reclaim), never a crash. posix_fallocate preallocate. 9/9 unit tests (real temp FS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
bff965a62e |
L3 3.0: shared per-node LMDB metadata index (CpL3MetaStore)
content_hash(32B)+payload_kind -> (disk,file,slot,page_bytes,crc,last_access,hit_count,flags). Validated topology (research C + multi-proc re-bench): owner ranks write durably (serialized on LMDB's write mutex, ~11x headroom over spill demand), all ranks read exists_prefix lockless (~1-3us, coherent). writemap=False (multi-process-safe). last_access/hit_count carry the replicated logical clock so L3 eviction selection is rank-uniform (3.3 consumes it). write_batch commits an object's entries atomically + durably (data->fsync->index->fsync ordering); iter_entries + reopen drive cold-rebuild; clear() is the flush_cache hook. 7/7 unit tests (venv lmdb 2.2.1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
b8ab180f1a |
L3 3.0: config (disks + rank->disk mapping + PLP gate)
CpL3Config (TOML/JSON): per-machine L3 disks (paths + budgets), backend, shared index location/size, and the rank->disk mapping. disk_for_rank balances M CP ranks over N disks (r % N; ranks share a disk gracefully — throughput is drive-count-bound) or honors an explicit map; fail-loud on length/range mismatch. probe_disk_plp: O_DIRECT write+fdatasync latency heuristic for the durability gate (non-PLP disk must fail loud when require_plp). No hardcoded paths. 11/11 unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
04c6793ca4 |
L3 3.0: on-disk blob format + disk-slot free-list allocator
Phase 3.0 (L3 disk durable floor) foundational primitives, pure + unit-tested: - 64B self-describing blob header (magic/version/payload-kind/content-hash/CRC), verified on every read (verify_blob) -> torn/bit-rot slots become a MISS, not garbage. - slot_bytes_for_page: header+payload rounded to the 4K O_DIRECT boundary. - CpL3SlotPool: O(1) free-list over fixed-size per-payload disk page-slots (the L3 analog of CpSharedL2PageAllocator's free list), fail-loud on double-free/OOB, with reset() for the flush_cache hook. NOT a ring buffer (eviction frees arbitrary slots). Rank-local (a rank stores only its owned pages; L3 eviction selection rides the shared LMDB replicated clock). 12/12 unit tests. Design: docs_internal/cp_hicache_l3_phase3_impl_design.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leavelet
|
85585fcf4b |
Fix CP HiCache writing_check deadlock at the source: rank-replicated symmetric attach
Replaces the interim ack_queue_len==0 band-aid with a structural fix so the perf skip is
SAFE rather than removed.
Root cause: the radix attach/defer/prune/split probes read `_node_host_write_pending`
(membership in ongoing_write_through/pending_host_backups, which are drained ASYNCHRONOUSLY
at per-rank ack completion), and `_cp_subtree_has_unprunable_state` also read the rank-local
`lock_ref`. So per-rank ack-drain timing leaked into the match_prefix truncation ->
cache_protected_len -> logical_len/len(value) -> the attach predicate. On some ranks the
prepared backup attached, on others it was dropped to the synchronous write_backup catch-up
(catch_up_all_layers draft=True), whose ack-append is per-rank. ack_write_queue then diverged
across CP ranks, so writing_check's ack_queue_len==0 early-return diverged: a strict subset
entered the writing_check_min all_reduce while the rest advanced to recv_requests' broadcast
on the same tp_cpu_group (8 ranks; dp_size==1 resets enable_dp_attention to False) -> NCCL
deadlock (prod hang: node_id=50713, all 8 GPUs 0% util).
Fix: introduce TreeNode.cp_backup_pending, a rank-replicated marker set at the rank-replicated
prepared-backup attach (_attach_prepared_cp_backup) and the guarded write_backup registration,
cleared only at the MIN-committed commit (_commit_pending_backup) / rollback. Rewire the CP
radix probes (_cp_node_split_still_pending, _split_node guard, _cp_subtree_has_unprunable_state,
_inc_hit_count) to read this marker instead of the drain-timed dicts, and DROP the lock_ref read
in the prune probe (its only cross-rank skew is the backup lock, now covered by the marker).
_node_host_write_pending stays the drain-timed accessor used only by writing_check / eviction /
the visibility check (_node_host_write_ready). With the radix-path reads replicated by
construction, len(value)==logical_len on every rank -> the prepared backup always attaches (or
rolls back) symmetrically -> the asymmetric catch-up never fires -> ack_write_queue is symmetric
-> the existing ack_queue_len==0 skip is all-or-none (no per-tick no-op MIN, best perf).
Correctness: the marker is cleared only at/after _commit_pending_backup, which runs only under
the MIN frontier, so the MIN remains the SOLE host-visibility gate (no KV-corruption class
reopened); the marker is strictly more conservative than the old dicts (stays True until commit).
Fail-fasts: double-attach raises; _split_node propagates the marker. hi_mamba unchanged (gates
on ongoing only, no async ack skip). Single-layer-draft (
|
||
|
laoyao0822
|
f0c6b892b6 |
Release CP draft-hidden buffers after EAGLE target prefill
CP draft shared-KV target prefill only needs the CP-local draft hidden side channel until draft prefill consumes it. The reused ModelWorkerBatch was left with capture_draft_hidden_states enabled, and LogitsProcessorOutput retained the extra draft_hidden_states tensor past that point. Restore the capture flags after target prefill, disable draft-hidden capture for draft prefill, and drop the consumed reference. Constraint: Large CP prefill batches are memory-sensitive; an extra hidden tensor held across the speculative step materially increases peak memory. Rejected: Leave capture_draft_hidden_states sticky across draft prefill | it can capture or retain hidden state outside the target-side CP-local side channel. Confidence: high Scope-risk: narrow Directive: Keep CP-local draft hidden as a target-prefill-only side channel unless the draft worker explicitly owns a new lifetime contract. Tested: g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_paged_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/speculative/test_eagle_worker_v2_cp_hidden.py Not-tested: full speculative integration/E2E workload. (cherry picked from commit f31ef2293ce0cdda2359a5f4713996d78b47f9df) |
||
|
laoyao0822
|
1d7149df1c |
Avoid remap-cache aliasing across tensor views
The CP shared-KV remap caches may receive tensor views that share the same storage but represent different logical page rows. Keying only by storage pointer and shape can reuse a stale remap for another view, corrupting cache-hit materialization. Use the actual tensor data pointer plus stride, storage offset, and version so different views and mutations are not collapsed into one cache entry. Constraint: CP shared-KV bs>1 cache-hit paths reuse small tensor views over shared backing tensors. Rejected: Clear the remap cache on every call | would avoid aliasing but add avoidable hot-path churn and hide the identity bug. Confidence: high Scope-risk: narrow Directive: Do not reduce the remap cache key back to storage pointer only; same-storage views are semantically distinct here. Tested: g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_paged_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/speculative/test_eagle_worker_v2_cp_hidden.py Not-tested: local pytest for CP runtime import is blocked by missing starlette in the local base environment. (cherry picked from commit 7360ef13565dbf21428e3b121112135a2955f913) |
||
|
leaveletandClaude Opus 4.8
|
9368c33b23 |
L2 prereqs for CP8DP2 + L3: scope HiCache consensus to the CP group + reset pooled allocator on flush
Two latent CP shared-KV L2 correctness fixes, landed BEFORE L3 (not in an L3 commit). Surfaced by a per-CP8-instance (CP8DP2EP16) scoping review. PREREQ-1 (CP-group scoping). The B1 commit/evict consensus collectives (writing_check ReduceOp.MIN, placement_digest MIN/MAX + its tp_world_size<=1 entry guard, drain_storage_control_queues, the evict/prefetch MINs, the flush barrier) AND cache_controller.prefetch_tp_group all derive from self.tp_group/self.tp_world_size, which was params.tp_cache_group. tp_cache_group equals the CP group ONLY at dp_size=1 (enable_dp_attention False -> tp_cpu_group, and attn_cp_size==tp_size -> _ATTN_CP==_TP) -- the sole reason B1 works today. Under CP8DP2 (DP attention, attn_tp_size=1) tp_cache_group is the size-1 attn-TP group, so every `tp_world_size>1` collective silently no-ops per rank and the placement assert self-disables -> divergent placement -> shared-slab corruption. Fix: for CP hicache (cp_size>1) scope self.tp_group to the CP cpu group (get_attention_cp_group().cpu_group -- already used for the slab-handle broadcast) + self.tp_world_size to its size. A no-op handle change at dp_size=1 (same group object); the intended fix at CP8DP2. The single init-point change propagates to every consensus collective + un-gates prefetch_tp_group + re-enables the placement assert. PREREQ-2 (flush reset). HiRadixCache.reset() cleared the radix tree + host pool but never reset CpSharedL2PageAllocator -> stale free list/ranges/committed after flush_cache (leak; the shared pool was never reclaimed). Added CpSharedL2PageAllocator.reset() (rebuild the per-slab free list all-free, drop ranges + committed, restore the freshly-built placement_digest) called from cache_controller.reset() after the ack queues are cleared. Safe: flush_cache is idle-gated (no in-flight backup/reserve). This is also L3's clear hookpoint. Validation: new test_reset_restores_freshly_constructed_all_free_state + 89/89 pool suite (torch-2.11 container) + import smoke. PREREQ-1 is a no-op at dp_size=1 (live no-regression confirmed on the next prefill restart); CP8DP2 correctness is by construction (CP-group membership verified) pending a 2-machine run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
laoyao0822
|
9549b268d7 |
Avoid allocating indexer state for shared NSA layers
Target-model skip-topk layers reuse the previous active layer's top-k indices and should not run local indexer modules. Centralize the layer-needs-indexer decision, skip constructing indexers on shared target layers, and skip their checkpoint tensors during load while keeping nextn/draft layers conservative for state safety. Constraint: index skip should reduce GPU memory in both prefill and decode without changing top-k propagation semantics Constraint: nextn/draft layers report shared top-k behavior but still need local indexer state safety Rejected: Loader-only filtering | parameters are already allocated during model construction Rejected: Dummy indexer modules for skipped layers | preserves most of the memory cost this change removes Confidence: high Scope-risk: moderate Directive: Do not reintroduce indexer execution on skip_topk target layers without proving prev_topk propagation and weight residency semantics Tested: remote g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/speculative/test_spec_utils.py test/registered/unit/configs/test_nsa_index_layers.py test/registered/unit/models/test_deepseek_index_skip_weight_loading.py -> 19 passed Tested: remote g0034 cjy-glm5-new py_compile for modified runtime files Not-tested: full GLM5 model restart memory delta measurement |
||
|
laoyao0822
|
187b700406 |
Keep speculative grammar traversal scalar-safe
EAGLE verification can pass torch scalar tensors through the speculative tree traversal before calling grammar backends. xgrammar/tvm_ffi requires Python int token ids, so normalize traversal indices and draft token ids at the boundary while leaving tensor storage unchanged. Constraint: xgrammar/tvm_ffi rejects torch scalar tensors for GrammarMatcher.accept_token Rejected: Coerce tokens inside every grammar backend | the invalid value originates in speculative traversal and should be fixed before backend dispatch Confidence: high Scope-risk: narrow Directive: Keep grammar backend calls scalar-Python typed; do not pass torch scalar tensors through accept_token Tested: remote g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/speculative/test_spec_utils.py test/registered/unit/configs/test_nsa_index_layers.py test/registered/unit/models/test_deepseek_index_skip_weight_loading.py -> 19 passed Tested: remote g0034 cjy-glm5-new py_compile for modified runtime files Not-tested: live decode replay after this commit |
||
|
leaveletandClaude Opus 4.8
|
9f7c193eb1 |
2b GC follow-on: drop dead CpSharedL2NodeMetadata.{committed, committed_payload_layers}
Tier-2-GC follow-on (design §9.12 flagged vestige). The metadata's `committed`
bool and `committed_payload_layers` dict were write-only residue of the option-A
quorum the GC removed: set at construction, validated in __post_init__, copied in
split() -- but their only content/True writer was the deleted
_commit_cp_shared_l2_layers_batched. Verified tree-wide there is NO live reader
(`committed` appears only in split's `committed=self.committed` copy;
`committed_payload_layers` only in __post_init__ validation + split copy +
construction). Commit consensus under B1 is the allocator's `_committed_objects`
(MIN frontier), so `metadata.committed` was a redundant second source of truth.
CORRECTION to the original flag: `required_payloads` is NOT a vestige -- it is
LIVE (read by cache_controller.load_cp_shared_l2, the evict path hiradix:2748,
and the readback payload planning). KEPT.
Removed: the 2 dataclass fields + their __post_init__ validation + the two split()
child-copies + the cache_controller reserve-construction kwargs + 4 test
constructions/assertions. No behavior change (fields were never read). Pool suite
88/88 (syh-dev-new/torch-2.11 env) + import smoke confirm the metadata dataclass is
now {logical_len, padded_len, page_size, object_ranges, required_payloads, object_key}.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
leaveletandClaude Opus 4.8
|
07c2b9bc57 |
2b.2: B1 collective-free shared-pool evict-to-fit + delete the #5 deadlock
Replaces the 2b.1b shared-L2 capacity-miss SKIP with a deterministic collective-free evict-to-fit, and deletes the #5 deadlock. This is the gate that makes --enable-cp-shared-physical-l2-hicache safe in a real run. evict-to-fit (_reserve_write_cp_shared_l2_evict_to_fit, hiradix): on a shared-L2 reserve capacity miss -- which raises identically on every CP rank over the replicated free list -- snapshot the plannable shared-L2 host leaves (excluding the node being backed up) in the replicated SLRU order (_cp_host_evict_key = Phase-1 logical clock + node.id total-order tiebreak), release the coldest one at a time (each a replicated free-list mutation via the proven _evict_cp_host_for_write_admission teardown -> evict_cp_host -> allocator.release) and retry the pooled reserve after each, stopping at the first fit. Fragmentation-correct (free coalesces in _return_range). Identical victim set + order + deterministic reserve => every rank evicts the same victims and stops at the same point (design Thm 1), so NO collective is needed. The snapshot is one-level by design (a parent promoted to a leaf mid-loop is not re-pushed; deep cascades skip-and-retry next tick -- missed identically on all ranks). Exhaustion -> loud rate-limited skip (transient: pinned/in-flight objects hold the pool), never a hang. #5 deadlock DELETED: stripped the synchronize_across_ranks all_reduce machinery (the `while len(heap) and not all_ranks_done()` ReduceOp.MIN host_evict_done_min closure + the param) from _evict_host_for_physical_slots -- never reached (no caller passed True) and the headline CP-deadlock shape. evict_host(num_tokens) still works (single-arg, equivalent non-sync loop). Opus adversarial review = SHIP (all 7 findings PASS: determinism [_cp_host_evict_key purely replicated, pin_expiry dormant under CP], teardown byte-equivalence + no double-free, no infinite recursion + clean abort-on-partial, snapshot safety [no ancestor/descendant among host leaves], bounded termination, clean #5 strip, flag-off path untouched). Tests: new test_eight_ranks_evict_to_fit_stays_identical (fill->miss->release-in-order->retry -> identical placement, exercises coalescing) + 88/88 pool suite (syh-dev-new) + import smoke. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
0d87bc576f |
2b dead-island GC: collapse pooled-L2 allocator to the B1 commit model
The 2b.1b adopt-then-strip left the entire option-A commit-quorum apparatus
orphaned: B1's writing_check ReduceOp.MIN frontier (mark_object_committed) is
the all-ranks-done consensus, so the per-(rank,layer,payload) gather quorum it
replaced is information-redundant (design sec 2.4 Corollary). This GC removes
the apparatus and collapses the CpSharedL2PageAllocator commit model to exactly
{_ranges_by_object, _committed_objects, _free_by_payload} -- precisely what
placement_digest() hashes -- so the cross-rank digest assert now covers the
whole model and the dead quorum is structurally unrepresentable.
Removed (all proven dead under B1, confirmed by a full tree caller-sweep +
opus adversarial review = SHIP):
- allocator: commit_layer, _has_full_commit, _expected_layers_for_object_payload,
adopt_reserved_range, set_object_required_payloads; fields _commits_by_object,
_expected_ranks, _adopted_ranges, _required_payloads_by_object,
_expected_layers_by_object, _required_payloads, _expected_layers; ctor args
expected_ranks/expected_layers/required_payloads; module fns
broadcast_cp_shared_l2_decision, gather_cp_shared_l2_{preflight,commits,commit}.
reserve/split_committed_object/_drop_object simplified to ranges+committed-bit.
- cache_controller: the 3 dead _commit_cp_shared_l2_* fns, 3 imports, all 5
option-A ctor params (cp_shared_l2_{cpu_group,source_rank,broadcast_fn,
preflight_fn,commit_fn}), the per-object commit-contract builder/setter/call.
- hiradix: allocator construction updated (3 args dropped); live _split_node
caller and _get_cp_shared_l2_rank_and_group@167 intact.
- tests: split tests migrated to mark_object_committed; 4 dead-symbol tests removed.
Lost __init__ payload-has-slab validation is covered by reserve()'s own
ValueError (fail-loud at first write). Default (flag-off) path unchanged: the
allocator is never constructed when enable_cp_shared_physical_l2_hicache=False.
Net -580 LOC. Validated: 87/87 pool suite (syh-dev-new, torch) + 31/31 core
classes locally + edited-module import smoke. Out of scope (flagged separately):
the CpSharedL2NodeMetadata.{required_payloads,committed_payload_layers} vestige.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
leaveletandClaude Opus 4.8
|
01029c8df4 |
2b.1b: B1 collective-free pooled-L2 write-through (default-off)
The B1 write-through: every CP rank runs the IDENTICAL deterministic CpSharedL2PageAllocator.reserve over the replicated event stream -> identical placement with NO broadcast/gather; commit rides the writing_check ReduceOp.MIN frontier (mark_object_committed) instead of a per-(rank,layer,payload) gather quorum. cache_controller.py: _reserve_write_cp_shared_l2 stripped to every-rank reserve (dropped preflight#2/rank0-gate/broadcast#3/adopt/missing-payloads; per-object contract kept MF3; idempotent abort-on-partial SF3); _submit_write_cp_layer_states drops the 3 gather-commit calls (per-layer D2H + all-payload done-gate MF1 + per-node ack kept); submit_write_cp_all_layer drops both fallback gathers (zero-owned keeps its ack MF2). hiradix_cache.py: 3-way merge of l2_pooling shared-L2 init (clean, 0-conflict; Phase-1 clock + E1 + reserve-budget preserved) builds the allocator/slab when the flag is on and passes it to HiCacheController WITHOUT collective fns. The #5 deadlock removed: shared_l2_capacity reserve-miss now SKIPS the backup (reactive, rank-uniform) instead of _evict_host_for_physical_slots(synchronize_across_ranks=True) -- the deterministic shared-pool evict is 2b.2. mark_object_committed wired at _commit_pending_backup (the MIN commit). New _cp_assert_placement_replicated (SGLANG_CP_HICACHE_PLACEMENT_ASSERT) in check_hicache_events: MIN/MAX-reduce placement_digest across the CP group, fail-loud on divergence (Theorem 1 runtime gate; rank-uniform entry, cannot deadlock). environ.py: SGLANG_CP_HICACHE_PLACEMENT_ASSERT (EnvBool, default off). Tests: 8-rank reserve-determinism (identical placement across 8 ranks + divergence-detectable) + full pool suite 91/91 + import smoke on g0033 syh-dev-new. TWO opus adversarial reviews both SHIP (cache_controller strip + hiradix wiring). NOTES: default-off (--enable-cp-shared-physical-l2-hicache); validated for write_through policy (prod default). Capacity-miss currently SKIPS (B1 shared-pool evict = 2b.2) so the flag must stay off in real runs until 2b.2. Dead gather island (3 fns + 3 imports, opus-confirmed inert) pending GC follow-up. Design+proof: docs_internal/cp_hicache_2b_pooled_l2_b1_design.md sec 2.4/9. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
50bc923ad6 |
2b.1b step 1: mark_object_committed (B1 MIN-driven commit)
Add CpSharedL2PageAllocator.mark_object_committed(object_key) -- the B1 commit path that marks an object committed directly, bypassing the per-(rank,layer,payload) commit_layer/_has_full_commit quorum. Under B1 the all-ranks-done consensus is the writing_check ReduceOp.MIN frontier (the same barrier that releases the node write lock), so every rank calls this with the same object_key at the MIN commit point and the committed set transitions rank-uniformly (feeds placement_digest). Resolves opus-review MF4: split_committed_object stays coherent for a B1-committed object (no per-layer commit facts) -- it adds children to _committed_objects and the empty _commits_by_object is harmless because B1 never calls _has_full_commit. 7 unit tests (pure-Python): commit-without-quorum, idempotent, unknown-raises, stat bump, digest reflects committed set + lockstep equality, split-of-marked-object coherent, release-after-mark frees pages. Full suite 88/88 green on g0033. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
7850aab1a2 |
2b.1a: placement_digest gate on CpSharedL2PageAllocator (B1 proof obligation)
Add CpSharedL2PageAllocator.placement_digest() -- a deterministic, order-independent SHA-256 of the full replicated allocator state (free list + per-object ranges + committed set). This is the B1 proof-obligation gate from the 2b design (sec 2.4, Theorem 1): the write-through (2b.1) cross-rank assert hashes this each tick and EQ-checks it across CP ranks, turning placement-determinism into a fail-loud runtime invariant -- any reserve/release fired on a subset of ranks or with a per-rank arg changes the digest. 6 unit tests (pure-Python, no torch): determinism, changes-on-reserve, identical-op-sequence => identical-digest, divergent-op => different-digest, reserve+release restores digest (free-list coalescing), placement-order sensitivity. Full suite 81/81 green on g0033 syh-dev-new. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
7de882b622 |
2b.0: adopt l2_pooling pooled-L2 mechanics (default-off)
Adopt the ownerless shared-physical-L2 mechanics from origin/l2_pooling onto the Phase-1 branch, behind --enable-cp-shared-physical-l2-hicache (default off -> behavior-neutral). Foundation for the B1 collective-free allocator (2b.1/2b.2) and the L3 durable floor (Phase 3); no coordination yet. - cp_shared_l2_pool.py (new): CpSharedL2PageAllocator (deterministic first-fit, capacity charged once, no x cp_size), hugetlbfs-2M slab primitives, position-indexed object ranges, commit-quorum data model. - memory_pool_host.py: SharedHostTensor* slab orchestration (3-way merged; fix-output's net change to this file is blank-line-only -> additions-only, no content lost). - server_args.py: flags + _handle_cp_shared_physical_l2_hicache_validation (3-way merged clean with Phase-1 assert + fix-output EnvField). - test_cp_shared_l2_pool.py: 75 unit tests (allocator/slab/quorum), green on g0033 syh-dev-new; envs stub provides SGLANG_REQ_WAITING_TIMEOUT (read by Phase-1 validation). Design + B1 proof: docs_internal/cp_hicache_2b_pooled_l2_b1_design.md (sec 2.4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
8f1e85a992 |
CP HiCache: replicated-clock SLRU eviction, collective-safe (Phase 1)
Make CP shared-KV HiCache eviction collective-free and scan-resistant by removing every per-rank wall-clock input from the eviction/admission decisions (any such input desyncs the must-be-replicated victim set / batch across the 8 CP ranks and can deadlock the collective-coupled writeback/load-back). - Replicated logical clock: TreeNode.next_access_time() (a process-global monotonic counter, mirrors mamba/swa's get_last_access_time) replaces time.monotonic() as the source of last_access_time at every radix bump site (radix_cache __init__/match/insert; hiradix CP match/insert/_insert_helper_host; reset() zeroes it). creation_time/pin_expiry intentionally stay wall-clock. Because match/insert events are replicated (reqs broadcast from rank 0 over the replicated tree), last_access_time is now identical on every rank. Unique ints also give a strict total order (no LRU tie ambiguity). No duration arithmetic reads last_access_time, so non-CP LRU is unchanged; mamba/swa use their own TreeNode and are untouched. - CpReplicatedSLRUStrategy = (is_protected[hit>=2], last_access_time, node.id): scan-resistant (cold one-shots stay probationary, evicted before reused/ protected prefixes), recency-within-segment ages out cold (not LFU), node.id is the deterministic total-order tiebreak (heaps never compare TreeNodes). Overridden for CP so it reaches every get_priority eviction site; the host write-admission key _cp_host_evict_key is switched from FIFO-by-id to it. - Co-fixes for the same per-rank-wall-clock class: pin_prefix is forbidden under CP (its pin_expiry victim-eligibility check is per-rank wall-clock; SLRU scan-resistance covers the benefit); the affinity head_age_s admission input is disabled (=0.0, relying on the replicated head_defer_count bound); and a server_args fail-fast guards CP shared-KV against SGLANG_REQ_WAITING_TIMEOUT>0 (its waiting-queue pruning is per-rank wall-clock). Tests: test_evict_policy.py adds TestCpReplicatedSLRUStrategy (scan-resistance, recency, total-order, full ordering) -- 30 passed in the dev-cu13 container; test_radix_cache_unit.py adds logical-clock determinism/total-order tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
8d73919d02 |
Gate CP HiCache write-backup on hard deficit, not the free-room watermark
A long cold-request workload flooded the prefill log (~177MB / 32K lines in 8min, all 8 CP ranks) with cp_host_reservation_plan_insufficient + prepare_write_backup_reservation_failed even though the host pool was <42% used and the writes physically fit. Root cause: _free_room_deficit folds the proactive 20% free-room target (hicache_host_free_room_ratio) into the admission deficit once a lane dips below the 10% trigger. _evict_cp_host_for_write_admission then failed -- and evicted nothing -- whenever it could not reclaim that full 20% target, which is impossible while the residual is locked/pending. The lane never drained, so every subsequent backup re-issued the same impossible demand and re-logged on every tick. The reservation failure is itself a graceful skip, so there is no crash; the symptom is the log storm. Gate write-backup admission on the HARD deficit -- max(0, required-available) over the target and draft lanes (draft_available is 2**62 with no draft pool, a no-op) -- and evict toward the watermark best-effort, admitting whenever the write itself fits. This drains the stuck lane and backs the request up instead of skipping+flooding. The proactive watermark stays as the eviction target (deficit_by_owner unchanged); it is no longer a hard admission gate. The decision is computed from rank-replicated state on the collective-free reserve path, so every CP rank makes the same admit/skip choice (opus-reviewed rank-safe). Also rate-limit the four host-reservation fallback warnings (once per 10s per fallback_name, with a suppressed count) so any genuine exhaustion degrades quietly instead of producing a log storm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
laoyao0822
|
96158fa110 |
Bound overlap prefill to one pending CP batch
CP shared-KV batch planning needs the immediately previous prepared batch to remain visible as a virtual prefix, but launching another batch before processing that result can leave multiple prepared plans racing against radix insertion. The event loop now processes the previous result after planning the current batch and before launching it, preserving one-batch lookback without accumulating deeper overlap state. Constraint: CP HiCache prepared batch views are inserted during process_batch_result, not at forward launch. Rejected: Process previous result before planning current batch | loses visibility of the previous pending prepared plan needed by current planning. Confidence: medium Scope-risk: moderate Directive: Do not increase non-PP overlap depth for CP shared-KV without adding pending-radix reservation semantics. Tested: Remote cjy-glm5-new pytest test/registered/unit/disaggregation/test_overlap_disagg_prefill_event_loop.py test/registered/unit/disaggregation/test_overlap_disagg_decode_event_loop.py Not-tested: Full E2E latency impact of reduced overlap depth. (cherry picked from commit 7df8723eee8203e9e334b229178e3f8bac61396a) |
||
|
leaveletandClaude Opus 4.8
|
512fe92a83 |
Fix CP HiCache catch_up_all_layers fallback on chunked-prefill final chunk
Under overlap scheduling a chunked request's final-chunk write-backup prepare read a stale `is_chunked` (>0): that per-tick counter is decremented in process_batch_result, which the overlap loop runs AFTER the run_batch prepare hook. So prepare floored `backup_end` to a page boundary (the intermediate-chunk rule) and dropped the now-complete final tail page, while the final non-chunked insert builds the radix node at full unaligned length. The exact-equality attach predicate (prepared.logical_len == len(value)) then failed by (num_tokens-1) mod page_size, dropped the per-layer overlap backup, and forced the serial all-layer catch-up (~89% of large chunked requests in prod). Decouple the floor decision from the stale counter: the scheduler marks `req.cp_backup_is_intermediate_chunk = (req is self.chunked_req)` in `_prepare_hicache_write_backups_before_forward` (the live chunked_req identity is the authoritative "will be chunked further" signal, set this tick before run_batch), and the prepare candidate builder floors on that flag instead of `is_chunked`. Intermediate chunks still floor; only the misclassified final chunk now backs up its full tail and attaches the overlap backup. Tests: update the chunked prepare test to the new flag; add a regression test that a final chunk with stale is_chunked reserves the full tail; add an offline repro that drives the real prepare/insert/probe and sweeps unaligned tails. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
2212963a6c |
Harden OpenAI tool-call/chat-template + reasoning parsing
serving_chat: - Tolerate a malformed historical tool_call `arguments` string (a valid JSON document followed by trailing content) instead of 400-ing the whole multi-turn request: salvage the leading JSON document via raw_decode, else keep the raw string. (A 112-message tool-history request was rejected with orjson "unexpected content after document".) - Catch TypeError (not only jinja2.TemplateError) from the chat-template render so a `tojson` filter on a Jinja Undefined becomes a clean 400 instead of a 500 (upstream #20700 / 5e9bd21979). reasoning_parser: - Strip only LEADING think-start marker tokens; a global replace would delete a `<think>` token that legitimately appears inside reasoning content. Preserve model-generated whitespace in reasoning/normal text (drop .strip()/.rstrip()) (upstream #24251 / dac78768f0). - Add Glm45 detector tests: leading-only strip, token-inside-content preserved, repeated leading markers, streaming trailing whitespace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
leaveletandClaude Opus 4.8
|
e23168e7f5 |
Fix CP shared-KV bs=1 cache-hit zero-prefix corruption; spans are sole compose input
A bs=1 cache-hit prefill produced deterministically wrong output (e.g. "0.5,0.5,0.5").
Root cause: regression
|