- http_server: chat + generate endpoints read x-request-id header into rid (the
Rust PD router drops the client body rid but forwards the header), so the
client id reaches forward_batch.rids for exact cross-send join.
- cp_hicache_trace.dump_tensors + SGLANG_NSA_DUMP_DIR: torch.save q/composed-KV/
selection/attn_out at layer 0 for rids starting 'dump-' (extend forwards), to
diff L1-hit vs L2-reload by relative error (beats fp-nondeterminism that
defeats binary hashes). Wired into nsa_backend flashmla_sparse path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a per-request content fingerprint (hash of extend input-ids + seq_len) to
fwd_hash so the SAME request forwarded from an L1-hit (known-good) and an
L2-reload (suspect) can be JOINED across the log without rid (the Rust PD
gateway strips the client rid). Gated to bs<=1 forwards (the join is only
meaningful single-request, and this skips the c=24 flood forwards so the
level-3 log stays small). The analyzer joins by ck and reports the first
DETERMINISTIC stage (topk/attn) that diverges = corruption localized.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The decisive level-1 instrumentation for the cross-rank logical-loc
divergence hypothesis. Prior traces all keyed PER-RANK (host pools aren't
comparable across ranks) -- which structurally hid the one invariant that
matters: the visible LOGICAL-loc sequence for a reloaded node must be
bit-identical on every CP rank, or the cross-rank gather reads token j from
its owner rank at a slot that rank filled with a different token.
- visible_locs_hash (cache_controller.load_cp): per-node, per-rank khash of
the visible logical locs, emitted for EVERY node incl. zero-owned so the
analyzer can diff across ranks by node_id. Divergence = root cause.
- victim_set (hiradix _evict_cp_load_back_owner_lanes): per-rank device
load-back eviction victim ids. Device eviction is keyed on wall-clock
last_access_time (per-rank, NOT replicated -- unlike host eviction's
deterministic (priority, node.id)); divergent victims is the trigger.
- release_draw (allocator.alloc_pages_with_owners): fires when a lane's free
bucket is exhausted and the deferred-free release bucket is tapped -- the
use-after-free/aliasing secondary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two metadata agents verified that all cache-hit-extend metadata (positions,
page_table, cu_seqlens, topk offset, MoE row-selection) is correct and
prefix-aware. The one reload-specific, unguarded thing is the per-page OWNER
replay: load_cp builds selected_logical_locs = node_device_indices[owned_positions]
and explicitly SKIPS re-validating that those pages are owned by this rank (perf;
the allocator's owner assert is gated behind debug_mode). A per-page owner
permutation (correct bytes, wrong logical page) passes the aggregate host-keyed
value-hash but makes attention read the right bytes for the wrong positions ->
reload-only repeated-token garbage.
Add owner_check (level 1): at each reloaded node, owner_for_logical_pages(
selected_logical_locs // page_size) must all == cp_rank; log bad_owner count +
valid/padded/tail_pad. bad_owner>0 = the per-page owner/padding permutation =
root cause, which the aggregate byte-hash cannot detect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both cached KV components round-trip byte-perfect, so the bug is downstream in the
cross-rank COMPOSE/GATHER on a reload forward. Add level-3 hashes targeting the
active suspect = materialize_prefix_and_reuse_current_kv_page_slots (reloaded
prefix gathered via modulo-owner IPC descriptors + fresh current via page_inverse
staging; the abutting prefix|current boundary is the suspect):
- gather_out: the live composed dense KV the attention actually reads (mixed_locs
gather), with prefix_pages/current_pages; nz==0 = zero/uninitialized composed KV
(the observed 0|0|0), self-contained.
- span_owner: per prefix/current span -> owner ranks (modulo) + physical pages +
logical pages + per-span hash/nz, to verify the prefix span maps to modulo
owners and catch a boundary conflation/off-by-one.
- ipc_desc: owner/src/logical-page ranges in build_cp_shared_kv_ipc_page_descriptors
(the modulo-owner prefix gather chokepoint).
- MoE stages (forward_deepep, via fwd_hash, valid-row local tensors only, never the
a2a-permuted intermediate): moe_postsel, router_logits, topk_ids/topk_w,
experts_out, moe_out_compact, moe_out_restored -> brackets select/router/topk/
dispatch-combine/shared-add/row-restore.
(The symm ComposePlan is dormant/unwired, so excluded.) All level 3, eager-extend
only, try/except-guarded, helpers khash/knz/rng. Analyzer flags zero composed KV +
zero spans + MoE-stage zeros.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Main KV round-trips byte-perfect yet output is garbage, so add the two
unverified components:
1. NSA INDEXER-K round-trip (level 2): hash the device index_k_with_scale_buffer
at backup (_backup_indexer_from_device_per_layer) and reload (NSA
load_to_device_per_layer, after _load_indexer), keyed by host-slot fingerprint
+ layer, with khash+nz. The indexer selects which tokens attention attends
(top-k); if it corrupts on reload -> wrong selection -> garbage even with
correct main KV.
2. FORWARD-side per-layer hashes (level 3, eager extend path only, cuda-graph
guarded): attn-input, attn-output (pre-residual), topk_indices (the indexer's
selection output -- direct consumer of the indexer-K), and MoE-input, in the
DeepseekV2/GlmMoeDsa decoder layer forward. Localizes where a reload forward
diverges: topk diverges => indexer-K cache; attn-out diverges (topk ok) =>
main KV/page mapping; moe-in diverges (attn-out ok) => residual/MoE.
Analyzer compares indexer reload vs backup (host_fp keyed) + flags zero/degenerate
hidden states per stage. Level 3 (SGLANG_CP_HICACHE_KV_TRACE=3) captures everything.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first KV-value pass couldn't compare (reload hashes keyed by the merged
load op's node id, backup hashes per node -> no join). Fix: emit the host-slot
range fingerprint at both backup and reload (host slots are the stable identity
across node id / splits / merge), and the analyzer matches on
(rank, layer, host_fp). Also add a nonzero-byte count (nz): real KV is never
all-zero, so an nz==0 slice is uninitialized/zero KV -- a direct, self-contained
corruption flag that needs no cross-stage matching. This targets the observed
'0|0|0...' garbage on large-cached reloads (store never filled the backed-up
pages, or the round-trip lost them). Gated at SGLANG_CP_HICACHE_KV_TRACE=2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The index trace proved per-node addressing is self-consistent, so the
corruption must be in the KV VALUES, not the bookkeeping. Add a full-tensor
position-weighted fingerprint (khash) of the actual device KV at two points,
both keyed by (node_id, layer_id) in owned-position order so they compare
directly:
- backup_kv_hash: device KV at backup, taken on the DEFAULT stream (outside the
write_stream block) so it is the correct POST-STORE value, not the racy
write_stream copy's view.
- reload_kv_hash: device KV after the H2D load (on load_stream, in-order after
the copy).
If a reload hash matches no backup hash for that node+layer, the round-trip
delivered wrong KV -- catching BOTH a per-layer store-vs-copy ordering race
(backup-correct != reloaded-stale) AND transfer corruption, in one run. Since
the compose/attention is identical fresh-vs-reload and stateless-per-forward,
all-ranks byte-correct shards => correct output, so this per-rank value check is
complete for the round-trip. Gated at SGLANG_CP_HICACHE_KV_TRACE=2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Heavily-forked CP HiCache produces KV corruption only after an L1->L2->L1
round-trip (eviction + reload), scaling with cached volume. Static reading and
upstream-diff are exhausted, so trace the data flow and check four invariants
directly, gated by SGLANG_CP_HICACHE_KV_TRACE (0=off, 1=structural lifecycle,
2=+compose/free/ack), off by default and cheap when off.
Trace points (keyed by node_id; rid_map ties client rid->node_id):
- backup_reserve / backup_d2h: host<->phys slots written (H2/H4)
- split: prefix+suffix partition vs parent (H1)
- evict: write_pending at device free (H4)
- reload_node: host slots read on reload (H2)
- reload_assign: reloaded device locs -> forward
- dev_free / write_ack: device free vs backup-complete ordering (H4)
- compose / remap: per-forward row-id cache reuse + unmapped count (H3)
New mem_cache/cp_hicache_trace.py (cptrace/rng helper; values rendered
space-free for offline parsing). The companion analyzer joins these by
node_id; the decisive, request-independent check is H2: any node whose reload
host-slot fingerprint differs from its backup fingerprint is the corruption
source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build_flattened_request_row_ids ran once per layer in the per-layer
prefill attention loop and rebuilt the indexer-seq-len-derived row ids
on the GPU via repeat_interleave with a *device* repeats tensor, which
forces a device sync (it reads sum(repeats) to size the output) and
serializes the launch thread every layer.
The row ids depend only on indexer_seq_lens_cpu, which is identical
across every layer of a forward pass. Add
get_cp_shared_kv_flattened_request_row_ids, which builds them once and
caches on the ForwardBatch (keyed by expected flattened length + device
so a different batch shape or the draft-vs-target pass never reuses a
stale tensor), and build on CPU then move once to drop the sync.
In a TP-5 warm cachebench trace this collapsed 1817 wrapper calls to 46
actual builds (97.5% fewer) and dropped ipc_materialize visibly.
GSM8K 200q x2 = 0.955 / 0.950, 0 invalid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EAGLEWorkerV2's target prefill always captured the global FULL hidden,
so the bs>1 NSA CP shared-KV draft extend received a global hidden whose
row count (real total tokens) matched neither the per-rank CP-local count
nor the MLP-sync CP-aligned padded count, tripping
[CP_SHARED_KV_FAIL_FAST][draft_batch_gt1_spec_hidden_shape_mismatch]
under SGLANG_ENABLE_SPEC_V2=1.
Mirror the legacy EAGLEWorker contract: add _can_use_cp_draft_shared_kv
and, when CP draft shared-KV applies, capture the CP-local hidden side
channel (CaptureHiddenMode.NULL + capture_draft_hidden_states) instead of
FULL. The v2 consumer already prefers draft_hidden_states when present
(commit 5e22279670 added the consumer but not this producer side).
Fixes the shape at the source rather than loosening the NextN fail-fast.
CP-off keeps FULL; non-CP / bs=1 unaffected; legacy v1 path untouched.
Validated on g0033 PD (prefill + 2-node decode): bs>1 up to 17 incl. the
former-crashing bs=6, 0 non-400 errors, GSM8K 0.965 / 0 invalid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deferred non-chunked inserts can happen while a CP HiCache per-layer backup is still in flight. Rolling back the unattached prepared backup at unfinished-request time drops the only reservation that final insertion can attach, so cache_finished_req has to start a post-forward backup and catch up all layers synchronously.\n\nKeep the prepared backup for non-chunked deferred inserts and continue rolling it back for chunked middle inserts, where the suffix may be extended by later chunks and the old backup would cover the wrong range. Document the failure mode so future changes do not rediscover the same fallback path.\n\nConstraint: CP HiCache radix split cannot mutate in-flight backup nodes.\nConstraint: Chunked prefill middle inserts still need rollback because their backup range is not final.\nRejected: Always rollback unattached backups | causes post-forward catch_up_all_layers for non-chunked deferred inserts.\nRejected: Always preserve unattached backups | can attach stale backup ranges for chunked middle inserts.\nConfidence: high\nScope-risk: narrow\nDirective: Do not clear req.cp_hicache_prepared_backup on non-chunked deferred insert without proving final insert no longer needs it.\nTested: remote cjy-glm5-new py_compile radix_cache.py\nTested: remote cjy-glm5-new pytest test_cp_hicache_metadata.py::{nonchunked preserve,chunked rollback} => 2 passed\nNot-tested: live ETE replay after restarting prefill with this exact commit
Decode DP dispatch was collapsing onto a few ranks because the controller only randomizes among exact minimum load pairs. The load snapshot undercounted decode handoff work: pending prefill-info requests were absent, and DecodeRequest wrappers in prealloc/transfer queues were skipped because their rid lives on .req.
This makes scheduler load accounting unwrap DecodeRequest items and include pending decode requests, so TOTAL_TOKENS sees queued handoff backlog instead of repeatedly treating busy ranks as empty.
Constraint: Do not mask imbalance with synthetic per-request token penalties; dispatch should be driven by accurate observed load.
Rejected: Add req*4000 or other queue penalties | heuristic, workload-dependent, and hides the accounting bug.
Confidence: medium
Scope-risk: moderate
Directive: Any new decode handoff queue must be included in get_load() or DP routing can regress to stale/min-load collapse.
Tested: Remote cjy-glm5-new: PYTHONPATH=python python -m pytest -q test/registered/unit/observability/test_scheduler_metrics_load.py test/registered/unit/managers/test_prefill_adder.py -> 27 passed.
Not-tested: Fresh decode ETE distribution after service restart.
The affinity scheduler needs to know whether the current batch is led by a chunked request. The rebase carried call sites that referenced an old private field name, while PrefillAdder only retained a boolean flag, causing startup failure before scheduling could run.
Constraint: CP bs>1 affinity must classify a chunked-led batch without reopening chunked-tail mixing behavior.
Rejected: Recreate the old private _chunked_req_in_batch attribute | keeps a stale name and hides the public state transition in PrefillAdder.
Confidence: high
Scope-risk: narrow
Directive: Keep chunked_req_in_batch and has_chunked_req_in_batch updated together when adding new chunked admission paths.
Tested: Remote cjy-glm5-new: PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_prefill_adder.py -> 26 passed; combined run with scheduler load accounting tests -> 27 passed.
Not-tested: Full ETE restart after this commit alone.
Revert the tail-chunk co-batching gate from 54c056af because allowing a continued chunk to share the next CP bs>1 batch reopens the mixed chunk/page-tail scheduler risks we are currently avoiding. Keep the independent real-prefix budget accounting so chunked requests still contribute their carried prefix to CP cached-token and buffer estimates.\n\nConstraint: Chunked-prefill requests must remain solo until the CP split/page-tail contract is revalidated for mixed batches.\nRejected: Full revert of 54c056af | it would also drop true-prefix budget accounting and under-estimate cache/buffer pressure for admitted chunks.\nConfidence: high\nScope-risk: moderate\nDirective: Do not reintroduce tail-chunk co-batching without tests covering page-tail split, CP buffer admission, and ETE chunked+cache-hit replay.\nTested: Local py_compile for environ.py, schedule_policy.py, cp_shared_kv_compose.py, test_prefill_adder.py, test_cp_shared_kv_compose_v2_8rank.py.\nTested: Remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/managers/test_prefill_adder.py -> 25 passed.\nTested: Remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 157 passed, 2 subtests passed.\nNot-tested: Full mixed replay with chunked-prefill traffic after service restart.
The syh rebase kept callers that expect reusable batch slot spans, while the restored CUDA IPC runtime lacked the helper and indexer still passed a symm-only writer-rank argument. Restore the batch-scoped span cache and remove the stale symm argument so the production compose path stays on CUDA IPC with exact per-request spans.\n\nConstraint: Production main-stream compose should use CUDA IPC, not symm writer-rank routing.\nRejected: Re-enable symm writer-rank parameters | benchmark showed no main-stream win and callers fail against IPC runtime contracts.\nConfidence: high\nScope-risk: narrow\nDirective: Keep slot-span metadata batch-scoped; do not rebuild Python span descriptors per layer without benchmarking.\nTested: Local py_compile for cp_shared_kv_runtime.py, cp_shared_kv_prefetch.py, nsa_indexer.py, nsa_backend.py.\nTested: Remote cjy-glm5-new pytest test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 157 passed, 2 subtests passed.\nNot-tested: Full ETE prefill/decode runtime after restarting services.
The syh branch still passed symm writer metadata into the RAGGED MLA
partial-current compose path. After rebasing the CUDA IPC staging path,
those keyword arguments no longer belong to the runtime contract and would
raise at runtime before reaching the IPC compose implementation.
Constraint: Main-stream CP shared-KV compose should use the measured CUDA IPC staging path; symm remains an experimental/benchmark path.
Rejected: Keep writer-rank arguments behind the symm env | the runtime signature has been restored to the IPC contract and the env-gated symm compose is not the production default.
Confidence: high
Scope-risk: narrow
Tested: python -m py_compile nsa_backend.py cp_shared_kv_runtime.py cp_shared_kv_prefetch.py nsa_indexer.py
Not-tested: Local pytest blocked by missing orjson in this environment; remote CUDA/ETE not run in this step
Expected no-prefetch paths were polluting production logs: no cache prefix, tiny/first-layer windows, and FP8 RAGGED top-k were being reported as fallback warnings. The prefetch contract now treats zero-prefix and first-layer misses as normal skips, while preserving warnings for non-zero misaligned prefixes and real consume misses after the first layer. The same change keeps RAGGED cache-hit prefetch eligible and records the CE/IPM prefetch contract in the plan doc.
Constraint: FP8 sparse prefill uses RAGGED top-k, but CP shared-KV prefix materialization is still page-slot based
Constraint: Layer 0 has no previous attention-window hook that can have prefetched the layer
Rejected: Warn whenever a prefetcher is absent | no-cache and too-short requests are expected synchronous paths and make logs unusable
Confidence: high
Scope-risk: moderate
Directive: Keep CP_SHARED_KV_FALLBACK warnings for unexpected contract failures only; use debug logs for expected skip paths
Tested: Local py_compile for cp_shared_kv_prefetch.py, nsa_indexer.py, nsa_backend.py
Tested: Remote cjy-glm5-new targeted regression: 3 passed, 21 warnings
Tested: Remote cjy-glm5-new full test_cp_shared_kv_runtime.py: 156 passed, 21 warnings, 2 subtests passed
Not-tested: New ETE run after prefill restart to confirm log volume reduction in production traffic
(cherry picked from commit e08e321e5929fdbb30102ec0b19c6ff0ecac7e7e)
CP shared-KV slot remaps already have forward-batch lifetime, but the IPC materialize path rebuilt owner/source/dense descriptor tensors on every layer. Cache prefix/current IPC descriptors on the token and paged slot-remap objects, keyed by layout, spans, device, descriptor kind, and prefix capacity, so all model layers can reuse the same request/batch-plan descriptors.
Constraint: Small-extend cache-hit workloads showed descriptor setup could exceed the all-reduce baseline before any IPC kernel work ran.
Rejected: Global descriptor cache | slot-remap lifetime is safer and avoids stale entries across request/batch-plan changes.
Rejected: Cache without physical page capacity | prefix descriptors encode capacity-invalid pages and must miss when capacity changes.
Confidence: high
Scope-risk: moderate
Directive: Do not reuse descriptors across different slot_logical_pages identity, CP layout, spans, device, or prefix capacity; stale descriptors can alias dense slots across requests.
Tested: Local py_compile; local git diff --check; remote g0034 cjy-glm5-new targeted descriptor tests 2 passed; remote full test_cp_shared_kv_runtime.py 146 passed, 21 warnings, 2 subtests passed.
Not-tested: Full ETE throughput/accuracy after descriptor cache; CUDA service benchmark still needed to quantify speedup.
(cherry picked from commit addd1ca1571e41458315d15304a0e841682fe8fa)
Cache-hit bs>1 current reuse can create very large dense attention buffers
while touching only a small set of current pages. The previous SGLang runtime
asked tai-kernel for a staging buffer sized like the full dense tensor, which
caused CUDA OOM before the current IPC fast path could run.
Switch token and index current IPC helpers to descriptor-compact staging: publish
the dense destination pages into compact staging slots and materialize peers
from compact source page ids back to the original dense destination pages.
Document the failure mode and the compact-staging contract so the dense-sized
contract is not reintroduced.
Constraint: CUDA + SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1 must fail fast instead of silently falling back to current-slot all_reduce
Rejected: Let staging allocation failure fall back to all_reduce | hides the bug and restores the expensive collective path
Rejected: Size staging by the full dense tensor | reproduces the 965MB staging OOM on long-prefix cache-hit batches
Confidence: high
Scope-risk: moderate
Directive: Current IPC helper source ids are compact staging ids; destination ids remain dense slot pages
Tested: Remote cjy-glm5-new PYTHONPATH=python:/mnt/beegfs/cjy/tai-kernel/python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 144 passed, 2 subtests passed
Tested: Local py_compile cp_shared_kv_runtime.py
Not-tested: Full ETE service restart with production traffic after this commit
(cherry picked from commit 906ecbe5d4f08b73242e98e2b628e26516d5b04a)
CP shared-KV cache-hit batches should compose long prefix pages and short current pages through page-slot IPC instead of falling back to dense all_reduce. Wire the runtime and prefetch consume paths to the TAI current-staging helpers, fail fast when the configured CUDA fast path cannot run, and document the bs>1 cache-hit benchmark evidence.
Constraint: bs>1 prefill must preserve the page-slot contract across fp8/bf16 and zero-lane current tails.
Rejected: Silent all_reduce fallback | hides correctness and performance regressions in production.
Confidence: medium
Scope-risk: moderate
Directive: Any future fallback in CP shared-KV CUDA fast paths must be explicit warning/fail-fast and covered by runtime tests.
Tested: Local py_compile cp_shared_kv_runtime.py and cp_shared_kv_prefetch.py; remote PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py (144 passed, 21 warnings, 2 subtests passed); remote TAI IPC benchmark fp8 bs>1 cache-hit matrix recorded in docs.
Not-tested: Full ETE mixed replay after replacing all current collectives with IPC.
(cherry picked from commit 8aa3b4ce59e0ebef5da5b0d07499a5f1d9785997)
CP shared-KV bs>1 compose must not silently fall back to dense full-buffer collectives when CUDA TAI materialize is expected. The fallback masks both correctness contract drift and severe synchronization/communication regressions, especially while comparing the symm path with the older IPC path.\n\nThis keeps CPU/unit-test fallback available, but makes production CUDA+TAI runs raise an explicit compose_v2 fail-fast for token-KV and index dense fallback. It also records the symm-vs-IPC comparison contract so barrier and collective counts are evaluated alongside elapsed time.\n\nConstraint: Production cache-hit-heavy bs>1 paths must expose unexpected dense collectives instead of silently taking them.\nRejected: Cherry-pick the old IPC branch wholesale | it conflicts with the symm compose design and would mix two transport protocols before benchmarking.\nRejected: Allow dense fallback with warning only | warning can be missed and still corrupts performance conclusions.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable dense full fallback in CUDA+TAI compose paths without a benchmark proving it is intentional and a correctness test covering cache-hit bs>1.\nTested: python -m py_compile for cp_shared_kv_runtime.py and test_cp_shared_kv_runtime.py; git diff --check.\nNot-tested: Remote container pytest/ETE; local pytest is not reliable in this workspace because dependencies such as orjson are missing.
- A chunked-led batch now seeds affinity_batch_warm_led from the CHUNK's
class instead of the first loop candidate: a 65536 first chunk is
cold-led (later colds are free density, caps bound them); latching
from a warm follower inverted cold-led-admits-everything and could
spuriously STOP the scan, under-filling the forward.
- The policy additionally requires no LoRA and no L3 hicache-storage:
their pre-classification continues make "first classified candidate"
drift from the true FCFS head, mis-routing the defer accounting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
专题 S4 (design: docs_internal/perf/prefill-compute-intensity-plan.md S4,
amended). Under FCFS a cold request joining a warm-led batch turns a
1-2s cache-hit forward into a 5-10s one, splitting the warm work into
the 新-cache-新 pattern. The policy prevents exactly that one thing:
- WARM candidates always admit (into a cold-led batch they are free
density — the cold extend dominates the forward anyway).
- COLD admits into an empty or cold-led batch (small colds co-batch
today; the FCFS head always starts a batch so the queue keeps moving).
- COLD into a WARM-led batch is skipped, bounded by a per-pass window
(W=16 skips), a head defer count (K=3 passes) and an age bound
(T=5s). On any bound the scan STOPS instead of force-admitting: the
cold waits for the same forward either way, but leads its own clean
batch next pass instead of polluting this one.
The skip is strictly post-match / pre-admit (after init_next_round_input,
before add_one_req): no lock, no allocation, no budget mutation to
unwind, and re-matching a skipped candidate next pass is exactly what
the scan already does after a cap rejection. Classification is the
in-scan match result (device prefix + host hit vs a 64-token floor) —
under FCFS+L2 no pre-scan signal exists, so this adds zero matching
work for inspected candidates. Disabled wholesale under priority
scheduling (the skip must not reorder across priority classes).
Three amendments vs the design draft, reasoned in the decision-table
docstring: cold+cold-led admits (STOP would regress today's small-cold
co-batching); starved heads STOP rather than force-admit (clean batch
boundaries at identical latency); priority interaction handled by
disabling rather than per-request comparison.
Decision logic is a pure function with table + bounds unit tests
(28/28 adder suite green). Default OFF.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
专题 S1 (design: docs_internal/perf/prefill-compute-intensity-plan.md
S1.0-S1.11). A batch containing a chunked prefill request has been
forced to bs=1 by the CP gate, so every chunk of a long prompt
monopolizes a forward while short-extend cache-hit continuations queue
— the direct cause of the replay TTFT tail (p90 19.3s / p99 49.2s at
91.8% cache hit). Yet mixed chunk batches already occur today (a
freshly-chunked request keeps earlier-admitted small requests), proving
the CP forward path is mixed-chunk-safe; only admission was asymmetric.
Three changes, the first flag-independent:
- add_chunked_req now seeds the budget with the chunk's TRUE prefix
(was 0), so the CP cached tally and the buffer estimator's mqa_logits
k_rows see the chunk's footprint before any later request is admitted
(landmine D1).
- New SGLANG_CP_PREFILL_MIX_CHUNKED (default OFF): with it on, the gate
admits requests after a chunked one and lets the existing CP caps
(extend / cached / buffer, now correctly seeded) bound the batch — a
FULL chunk still ends the scan by consuming the chunk-clamped extend
cap; only a tail chunk leaves headroom. A chunked prefix that is not
page-aligned (rare sub-page final-chunk tail) keeps its batch solo
(the CP page-aligned split would fail-fast otherwise).
- The symm staging capacity identity (admission extend cap + request
slack == staging pages) is asserted when the flag is on, locking the
coupling the design relies on (plan doc S1.4 I2).
Tests: 4 new adder units (budget seeding; tail chunk admits followers;
full chunk solo by budget; non-aligned prefix solo); the 8-rank
byte-exactness scenario gains a chunk-shaped request (2048-token
page-aligned carried prefix + 512 extend) — all four phases
(legacy/v2/symm/prefetch) byte-identical on g0033. Known pre-existing
cross-file pollution noted in problems.md P16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fingerprint hashes the whole source tree, so ANY code difference
between the prefill and decode trees tripped it — and it raised inside
try_ensure_parallel_info on the DECODE scheduler's event loop, crashing
the decode cluster. In practice that means a prefill-only restart with
a one-line fix (today: the inflight liveness fix) kills decode.
The checks that actually guard the KV transfer contract (page size,
kv cache dtype) remain hard errors; the fingerprint is now a warning
that still surfaces both hashes and source roots.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
E2e caught one request wedged FOREVER in disagg_prefill_inflight_queue
(probes 21s apart with zero traffic both showed #inflight-req: 1, no
reap/timeout warnings ever logged). Mechanism, established by reading
the full state machine: prefill Success is set locally by the transfer
worker on the LAST chunk; if the decode peer is torn down between the
handshake and the prefill's final send(), add_transfer_request silently
drops the chunk (no transfer destinations) — Success becomes
unreachable. The only external rescue, the decode ABORT notification,
is best-effort (silently swallowed on send error, no-op if it races the
room registration), there is no prefill-side heartbeat of decode
sessions, and the sender's only timeout covers Bootstrapping — the
inflight queue itself has no liveness bound. The orphan pins the
request's KV pages and rides every poll collective.
Two fixes, both reaped through the existing Failed branch via the
CP/TP MIN-reduce poll consensus (Failed=0 wins, so one rank concluding
flips every rank together — rank-uniform by construction):
- add_transfer_request: a room with no transfer destinations that is
NOT already Success (the dummy-rank handshake marking) now concludes
Failed loudly instead of dropping the chunk silently.
- Inflight residency timeout: entries stuck in a non-terminal poll
state past SGLANG_DISAGGREGATION_INFLIGHT_TIMEOUT (default 300s,
matching the sibling BOOTSTRAP/WAITING timeouts) get sender.abort()
and reap on the next poll. Covers what the hardening cannot: lost
ABORT datagrams, decode crashes.
Known sibling gaps left for follow-up: the decode transfer queue has
no Transferring liveness bound, and an abort that matches no queue is
still a silent no-op (much narrower race than first thought — work
requests are ordered before control requests within a tick).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First e2e launch crashed every CP rank at the symm token fill:
index_copy_cuda is not implemented for Float8_e4m3fn — the production
KV pool dtype, which the 8-rank test missed by building its pools as
uint8. The fill is a whole-token-row copy, so it is dtype-agnostic:
both the staging span and the current rows now go through uint8 views.
The 8-rank test now builds the KV pools and current rows as
float8_e4m3fn (payloads constructed as bytes, compared as bytes) so
the production dtype is what every phase exercises.
Validated on g0033: all four 8-rank byte-exactness phases under fp8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Opus review of the symm prefetch wiring found a latent group deadlock:
the prefetch hit path never touched _agreed_tai_ipc_peer_ptrs, so the
first use of a layer's pool tensor — which issues a capability MIN
all-reduce plus an IPC handle all-gather — only happened on the sync
fallback. With every rank hitting from batch one (the prefetcher runs
a layer ahead), a later lone-rank miss would be the only rank issuing
those collectives. Both consumes now seed the (idempotent, per-pool-
tensor-cached) agreement at entry, before any per-rank early return;
the index consume gets the pool buffer from its indexer call site.
Unreachable in today's config (prefetch env off) but a blocker for
ever enabling it.
Also per review: the [pool|staging] combined pointer-table cache now
keys per pool table instead of holding one — the token and index pool
tables alternate every layer and evicted each other, so the cache
never hit; and a debug-gated check that current locs all map into the
staging page inverse (index_copy_ has no skip semantics for stray -1).
Validated on g0033: 151 unit tests; all four 8-rank byte-exactness
phases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bs=1 MLA/index prefetchers replaced their consume-side trailing-range
NCCL all-reduce with the staging exchange: fill current rows straight into
this round's staging span (token-KV collapses to one cached index_copy;
the index fill kernel is just pointed at the staging page inverse),
cp_symm_barrier, then gather ALL current pages — this rank's own included
— from the stagings into the prefetched dense buffer. The symm+prefetcher
FAIL_FAST is gone.
Rank-uniformity moves with it: staging registration now also happens in
maybe_create (batch-logical gates, before any per-rank miss can diverge),
because with a prefetcher active the sync compose runs only on per-rank
misses and its lazy collective registration would hang. A hit/miss
divergence itself stays barrier-safe — both the prefetch consume and the
sync-compose fallback execute exactly one begin_round + barrier per
(layer, kind), and the counting barrier is shape-free (unlike the AR pair
it replaces, which would shape-mismatch).
Found by the new index test phase: the fill/remap kernel family skips
page id 0 as the SGLang dummy page, so a 0-based first staging slot was
never written. The staging layout now reserves row 0 (slot of current
page i = i + 1) for every kind, matching the convention instead of
depending on per-kernel behavior.
Launch-path cost: per-(kind,parity) peer pointer tables and the
[pool|staging] concatenations are precomputed/cached (identity pinned by
holding the pool-table reference); all prefetch descriptors, staging row
indices, and mixed_locs are built once per batch.
Validated on g0033 8xH200: 151 unit tests; 8-rank byte-exactness for
token sync symm (8 layers), index sync symm (4 layers, new phase), and
MLA + index prefetch consume_prefix_with_current vs the legacy sync
compose.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The publish-variant staging exchange measured ~equal to the default
compact-current AR (90.4 vs 86.5 ms/batch on the traced scenario): the
publish copy and the barrier serialized behind the 0.65 ms prefix
gather ate the transport win the isolated current exchange showed
(0.196 vs 0.354 ms). Fix the structure instead of the copy: current
rows are now written straight INTO the staging — the fill kernels take
their write destinations solely from page_inverse, so a per-batch
staging-remapped page inverse on the plan retargets them with zero
kernel changes — then cp_symm_barrier, then ONE slot-dense gather
covers prefix pages (pool pointers) and ALL current pages (staging
pointers, including this rank's own) through a concatenated 2*cp
pointer table where current slots carry owner = cp_size + writer and
src = staging slot. No publish copy, no prefix pre-gather, no second
gather.
The fused fill's loc outputs are dense-geometry-bound, so the token-KV
path computes mixed_locs/staging row indices once per batch (they are
layer-invariant) and the per-layer fill collapses to a single
index_copy_ into the zeroed staging span.
Benchmark (g0033 8xH200, byte-exact, idle-checked): 62.8 ms/batch vs
84.4 default Step A (-26%) and 60.8 ideal; publish variant was 88.0.
151 unit tests; 8-rank GPU byte-exactness vs v2 across 8 layers,
arena on and off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Peers only ever read the CURRENT pages of a rank's compose output — the
prefix comes straight from the IPC-registered KV pool — so the symm
region does not need to hold the whole dense buffer (pool-bound ~2.5 GB
double-buffered slab). It now holds one round of current pages in
merged-span order (extend-cap-bound, ~58-100 MB), and dense buffers
become purely rank-local (plain allocations or the optional local arena;
COMPOSE_SYMM no longer requires COMPOSE_ARENA).
Exchange per compose call: publish my written current pages
dense[page] -> staging[slot i] (slot = the page's batch current index,
identical on every rank, so peers address each other's staging with no
per-batch handshake), cp_symm_barrier, gather peers'
staging[writer][slot] -> dense[page] via the existing src!=dst page
gather. Reuse safety keeps the parity-half distance-2 argument, now on
the staging. Capacity sizing comes from the admission caps
(max_total_extend_tokens / max_batch_requests) with a pool-derived
fallback and the SYMM_HEAP_MB override; overflow fails fast
(batch-logical, hence rank-uniform).
Idea credit: laoyao0822's touched-pages-proportional staging
(906ecbe5d4), rebound onto our barrier-gated, group-agreed transport.
Validated on g0033 8xH200: 151 unit tests; 8-rank GPU byte-exactness
vs compose_v2 across 8 layers (arena on and off, parity halves
exercised); benchmark path e (real protocol) byte-exact, current-page
exchange 0.196 ms vs 0.354 ms compact-AR isolated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Radix insert can reach the pending-backup/stale-tail split probe on only a subset of CP ranks. The previous opportunistic ack drain called writing_check(), which may issue a scalar write-visibility collective while other ranks are polling inflight transfer state with a different tensor shape. That ordering mismatch matches the observed Gloo 4-vs-2 abort after CP_HICACHE_FALLBACK insert_deferred_pending_backup_split logs.
This keeps the split probe local and conservative: it leaves ready write acks queued and lets the globally ordered scheduler/cache-event path drain them. The regression test locks the contract by making any collective from the pending-split probe fail.
Constraint: CP radix insert/pending-split probing is not globally ordered across ranks.
Rejected: Drain ready write acks from the split probe | can interleave with scheduler poll collectives and corrupt distributed collective ordering.
Confidence: high
Scope-risk: narrow
Directive: Do not call writing_check() or any CP/TP collective from radix pending-split probes; drain write acks only from globally ordered paths.
Tested: Remote cjy container: PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 126 passed, 21 warnings.
Tested: Remote cjy container: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py => PY_COMPILE_OK.
Not-tested: Full ETE prefill/decode replay after this change.
Co-authored-by: OmX <omx@oh-my-codex.dev>
From the nsys CPU-gap attribution (launch thread, one 78-layer forward:
374ms API time; 642 cudaStreamSynchronize blocking 89.5ms and overlapping
122ms of the 505ms GPU idle; ~44ms pure-Python before concat_mla_absorb_q):
- memory_pool_host: skip validate_page_aligned_token_indices on CUDA
tensors in _get_indexer_page_indices and _prepare_load_page_indices —
torch.any/torch.equal there cost a queue-deep cudaStreamSynchronize per
layer-group submit (~0.42ms each, ~12.7ms/forward measured). Same
construction-based-invariant guard the CacheController pair check
already documents; CPU/test tensors stay validated.
- nsa_indexer: per-batch _CpRaggedIndexPlan replaces the per-F-layer
rebuild of the O(total-q-tokens) topk offset list and the 6-7 int32
ragged descriptor tensors (segment records, kv_lens/q_starts/q_lens/
k_bases/q_bases/current_bases). All inputs are batch metadata; the plan
is anchored on the forward batch with a content key over cp_index.
- nsa_indexer forward_indexer: read seq_lens_cpu instead of a device
seq_lens[i].item() per request per layer (one stream sync each).
- cp_shared_kv_runtime: get_or_build_batch_slot_spans caches the
layer-invariant prefix/current slot spans per batch (the builders read
logical_pages only for its shape); nsa_backend x3 + nsa_indexer call
sites switched.
Microbenchmark (idle H200, traced batch shape bs=12 / 44.6K q tokens,
test/manual/bench_cpu_gap_fixes.py, equality-checked): validator path
197.1us -> 59.2us per submit under a busy queue (x3.3); ragged plan
3238.6us -> 36.2us per layer (x90, ~128ms launch-thread time per forward
at 40 F-layers); slot spans 20.1us -> 0.5us (x41). Layer suites A/B vs
HEAD: identical failure set (5 pre-existing CPU-tensor indexer tests),
no regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses the opus review of the Step A/B compose stack:
- FAIL FAST on symm + prefetcher coexistence: the MLA/index prefetchers
issue their own per-span collectives and bypass the symm exchange, so
letting them coexist would make COMPOSE_SYMM a silent no-op on every
prefetch hit (measured "parity" that never ran). maybe_build_current_
page_writer_ranks now raises when a prefetcher is attached or
SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH is set.
- Arena parity now derives from a monotonic round epoch instead of raw
layer_id: EAGLE draft layers reuse decoder layer ids, so id-parity would
stop alternating halves (draft L0 -> next forward target L0 lands on the
same half) and repeated-id rounds would keep appending into one half.
begin_round(layer_id, kind) starts a new round (other half, offsets
reset) when the id changes OR a kind repeats; regression test included.
- Per-batch writer-ranks cache on the forward batch and a presence flag in
the plan key: previously the ~bs x current-pages writer list was rebuilt
AND tuple-hashed on every layer per call site, pure launch-path waste.
- Rank-uniformity invariant documented at _symm_exchange_current_pages
(any per-rank gate must go through a group agreement first) and an
actionable arena-overflow message naming SGLANG_CP_SHARED_KV_SYMM_HEAP_MB.
Deferred follow-ups (documented in the design doc): factor the two near-
clone v2 helpers, single barrier per F-layer (~308us/batch), persistent
device-side peer-ptr tables for the gather.
182 targeted tests + 8-rank GPU byte-exactness (v2 and symm) re-validated
on g0034 syh-dev-new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the last remaining NCCL collective in the bs>1 compose layer loop
(the compact-current all-reduce) with a DeepEP-style counting barrier plus
a CUDA-IPC gather from peers' symmetric dense buffers, behind
SGLANG_CP_SHARED_KV_COMPOSE_SYMM (+ COMPOSE_ARENA, both default off).
- CpComposeArena.register_symm: fixes capacity at the pool-derived bound
(logical pages x dense page unit, overridable via
SGLANG_CP_SHARED_KV_SYMM_HEAP_MB), allocates slab + flags in CUDA-IPC
memory, exchanges handles once over the CP group; deterministic bump
carve means peer_base + my_offset addresses any peer's dense buffer with
no per-layer handshakes. Registration happens only in the token-KV
compose (uniform first-use point); growth after registration raises.
- Current pages are single-writer at page granularity under the
page-aligned in-seq split, so the exchange is the existing
gather_cuda_ipc_peer_pages with src==dst page ids and writer (compute
owner) ranks; writers are built per batch by
build_batch_current_page_writer_ranks and gated by
maybe_build_current_page_writer_ranks (page_aligned metadata required).
The barrier runs even with zero remote pages (counts must match).
- _agreed_tai_ipc_peer_ptrs: the per-rank IPC capability probe is now
agreed across the CP group (one-time MIN all-reduce per pool tensor) so
ranks can never split between gather and collective paths and deadlock
on mismatched NCCL shapes.
- ComposePlan cache re-anchored ON the slot_remap object (forward-batch
lifetime) instead of a module-level tensor-identity key, which could go
stale when a freed tensor's address is reused by the next batch.
Validation (g0034 syh-dev-new): tai-kernel cp_symm_barrier correctness
(200 adversarial iterations, rotating 10ms producer delays, phase-safety,
flags drained at quiescence) and perf (7.7us max-rank latency) both pass;
8-rank GPU test extends to the symm path - byte-identical to compose_v2
across 4 layers (parity halves exercised, slab registered); mem_cache
suite 464 passed with the only failures being a documented pre-existing
sys.modules stub pollution pair, reproduced identically with
SGLANG_CP_SHARED_KV_COMPOSE_V2=0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bs>1 partial-current materialize issued one sum-all-reduce per request
span per buffer per layer (48 collectives per F-layer at bs=12, 2880 per
batch, 419ms + 49ms launch gaps in the production trace), all inline on
the compute stream. The data is a partitioned gather, not a reduction:
every byte has exactly one producer.
compose_v2 (SGLANG_CP_SHARED_KV_COMPOSE_V2, default on) replaces this with:
- fast path: one tai-kernel CUDA-IPC slot-dense gather covering ALL prefix
spans (full-range descriptors, -1 sentinels zero-fill current slots and
replace the dense zero-fill) + ONE collective over the compact current
pages (uint8 byte view; exact because every byte is writer-exclusive).
- fallback (no peer IPC): local materialize of all prefix spans + ONE
whole-buffer sum-all-reduce (rows are still writer-exclusive pre-reduce).
IPC capability is decided once by the cached peer-pointer probe; after a
successful probe a failing gather raises (no per-call try/except).
cp_shared_kv_compose.py adds the per-batch ComposePlan descriptor cache
(layer-invariant, keyed on the slot_logical_pages identity) and the
CpComposeArena with tier-S carve discipline (deterministic bump, layer-
parity halves; default off) so the Step B symmetric-memory conversion is
a registration flip.
Microbenchmark (g0034 8xH200, traced 12-req batch, per batch): per-span
214ms -> fused AR 119ms -> IPC prefix + compact current 84ms; symm target
61ms. Validation: 143 unit tests incl. v2-contract twins, legacy siblings
and rank-merged simulations under both paths; mem_cache dir 432 passed;
8-rank GPU byte-exactness vs legacy with real NCCL + IPC (test/manual/
test_cp_shared_kv_compose_v2_8rank.py) passed with no fallback markers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reserve_write_cp undrained-ack refusal returned a bare
HiCacheWriteFailure, which _reserve_write_cp_indices_no_collective
interpreted as a host-capacity failure: with free host space the
retry path tripped the predicted-no-deficit RuntimeError (crashing
the scheduler in exactly the scenario the gate exists to handle
gracefully), and with a deficit it triggered pointless evictions
before refusing again.
Give HiCacheWriteFailure an explicit reason (default host_capacity
keeps all existing constructors/semantics); the wrapper returns
non-capacity failures immediately as skip-this-round, and a
wrapper-level test pins that undrained_ack reaches neither the retry
admission nor host eviction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
node_has_undrained_write_ack scanned ack_write_queue on every
reservation, including the per-request-per-chunk prepare path whose
node ids are freshly minted and can never be queued. Track the max
node id ever appended to the queue: any queued id is <= max by
construction (no monotonicity assumption needed for correctness), so
fresh ids exit on one integer compare and the scan remains only for
re-reservations of old ids (the rare write_backup fallback).
All three ack_write_queue append sites now go through a single
_append_write_ack funnel that maintains the max — the zero-owned-rank
and non-CP write acks previously would have bypassed it, allowing
false negatives on ranks owning no pages of a node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dfa168abe9 made duplicate ack completion idempotent at the consumer
(writing_check), which is correct and lock-balance-safe but masks the
producer invariant breach: two acks can only coexist for one radix
registration when an ack is orphaned in ack_write_queue after a
rollback cleared ongoing_write_through/pending_host_backups,
re-opening the _node_host_write_pending guard for a fresh
registration. _rollback_pending_backup (write_backup's exception
path) was the one rollback that did not scrub the node's acks — and
it also left a half-submitted layer-write state alive, which later
forwards would keep driving, writing D2H into host slots the rollback
had already evicted.
Close the invariant structurally:
- reserve_write_cp refuses (HiCacheWriteFailure, the existing
skip-this-round path both callers already handle) when the node
still has an undrained final ack, computed by scanning the small
ack queue — no new state to keep in sync.
- _rollback_pending_backup now cancels the pending layer-write state
(write-stream sync so in-flight per-layer copies finish before the
host slots are evicted) and scrubs the node's queued acks via the
scrub factored out of _rollback_prepared_cp_backup.
- The consumer-side duplicate guard from dfa168abe9 is kept as
defense in depth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CP per-layer write-through can leave duplicated ready ack ids in the
ack queue while pending-split insertion drains write visibility. The
pre-scan observed both duplicates as registered before the first
completion removed radix state, so the second completion raised KeyError
and killed the scheduler.
The write-check path now records completed ack ids until matching queue
entries are drained. Duplicate completed acks are removed with a warning,
while genuinely unknown or unregistered acks still fail fast.
Constraint: CP HiCache pending-split drain may call writing_check outside the normal idle event path.
Rejected: Blind pop(..., None) for all unknown acks | would silently hide truly unattached or corrupt ack state.
Confidence: high
Scope-risk: moderate
Directive: Do not remove the completed-ack short-term memory unless duplicate and stale ack queues are proven impossible across TP MIN synchronization.
Tested: Local py_compile for hiradix_cache.py and test_cp_hicache_metadata.py.
Tested: Remote cjy-glm5-new py_compile and full test_cp_hicache_metadata.py, 121 passed.
Not-tested: Full ETE prefill workload after restarting service.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Over-long inputs produced two different client errors depending on
which bound rejected them: the TokenizerManager pre-check (raw
context_len) returned 413 PayloadTooLargeError ('The input (N tokens)
is longer than the model's context length (M tokens).'), while inputs
between that and the scheduler's stricter effective limit hit
validate_input_length and returned 400 BAD_REQUEST with different
wording (and a confusing 'X exceeds X' message since the check is >=).
Unify on the 413 format end to end:
- validate_input_length wording now matches the TokenizerManager
message, reporting the effective per-request limit.
- set_finish_with_abort takes status_code/err_type; the scheduler
length-rejection sites abort with REQUEST_ENTITY_TOO_LARGE +
PayloadTooLargeError. The batch handler previously queued the
over-long request WITHOUT marking it aborted (it proceeded to
prefill) — also fixed.
- Non-streaming aborts with 413 raise PayloadTooLargeError (now a
ValueError subclass so raw /generate-style endpoints that only
catch ValueError still respond; the OpenAI layer's except clause
is reordered to win and emit the 413 format).
- Streaming abort responses prefer the scheduler-provided err_type
over the HTTPStatus name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'[CP_PER_LAYER_TRANSFER] registered room=... chunk=...' fired at INFO
once per room per chunk on every prefill send. The one-time manager
registration stays INFO and failure paths stay WARNING; the
success-side finish log was already debug.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Absorb PR 11's final Para compatibility surface as an opt-in OpenAI serving layer rather than hard-coding business defaults into protocol models. The change adds server args for Para chat defaults, Kimi/GLM compatibility, tool-choice normalization, tool-role text flattening, and streaming first-chunk error preflight while preserving default upstream behavior unless explicitly enabled.
Reasoning token usage is also propagated through chat/completion usage paths, with GLM compatibility emitting completion_tokens_details.reasoning_tokens. Low-risk protocol fixes accept string image_url content parts and preserve GLM function-call argument value whitespace.
Constraint: Online Para-compatible deployments require request/response semantics that differ from default OpenAI serving behavior.
Constraint: Current CP/HiCache/bs>1 work must not be coupled to OpenAI serving compatibility changes.
Rejected: Merge PR 11 history directly | intermediate commits briefly hard-code chat max_tokens=32768 before later gating it by server args.
Rejected: Enable Para compatibility by default | would change non-Para OpenAI-compatible deployments.
Confidence: high
Scope-risk: moderate
Directive: Keep Para-specific serving policies behind explicit server args unless the business contract changes globally.
Tested: PYTHONPATH=python:. python -m unittest discover -s test/registered/unit/entrypoints/openai -p 'test_para_serving_protocol.py' -v (19 tests OK)
Tested: python -m py_compile modified OpenAI serving, tokenizer manager, server_args, function-call detector, and test files
Not-tested: Live router/prefill/decode OpenAI serving E2E after enabling Para flags.
Co-authored-by: OmX <omx@oh-my-codex.dev>
Per-layer prefill-to-decode transfer now finishes per request/rank/chunk, so success-path INFO logs can dominate production logs and hide actual failures. Keep successful finish breakdown and completion messages at DEBUG while preserving nonzero finish status as WARNING.
Constraint: Per-layer transfer is a hot path under CP shared-KV and may produce many batch completions per request.
Rejected: Disable CP per-layer transfer logging entirely | failures still need visible warning-level evidence.
Confidence: high
Scope-risk: narrow
Directive: Do not promote successful per-request transfer completion logs back to INFO without rate limiting.
Tested: PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_cp_per_layer_transfer.py::TestPerLayerTransferContext::test_successful_finish_does_not_emit_hot_path_info_log
Tested: python -m py_compile python/sglang/srt/disaggregation/cp_per_layer_transfer.py python/sglang/srt/disaggregation/mooncake/conn.py
Not-tested: Full local disaggregation suite blocked by missing local orjson dependency.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV bs>1 cache-hit loads already merge request load ops, but the host pool still rebuilt layer-invariant mapping work from the same host/device indices. Introduce a PreparedLoadDescriptor lifecycle around begin/end load, wire MLA KV and NSA index H2D loads through tai-kernel prepared submit when available, and add timing hooks plus regression coverage for descriptor reuse and explicit fallback logging. Record the P4/P6b design and benchmark results in the advanced feature notes.
Constraint: Radix residency and allocator decisions remain synchronous; only the data-transfer descriptor is prepared for per-layer async submit.
Constraint: Production fast path must not silently fall back when tai prepared H2D support is missing.
Rejected: Cross-batch descriptor reuse | descriptor lifetime and tensor ownership are only safe within one load operation.
Rejected: Change L2->L1 scheduling to layer-ahead prefetch in this commit | that is a separate lifecycle change after descriptor reuse is stable.
Confidence: medium
Scope-risk: moderate
Directive: Keep LayerDoneCounter per-layer readiness semantics; do not replace with all-layer waits.
Tested: python -m py_compile python/sglang/srt/mem_cache/memory_pool_host.py python/sglang/srt/managers/cache_controller.py
Tested: Remote g0034:cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py (88 passed)
Tested: Remote tai-kernel prepared descriptor CUDA test (6 passed) and P4 benchmark full matrix (90 rows)
Not-tested: ETE replay/GSM8K cache-hit correctness after this commit
Not-tested: Layer-ahead L2->L1 prefetch scheduling
Co-authored-by: OmX <omx@oh-my-codex.dev>
When chunked prefill is active, CP shared-KV bs>1 cannot consume more extend
tokens than the current chunk budget. If the CP-specific extend-token limit is
omitted, default it to rem_chunk_tokens so scheduler admission reflects the
reachable chunk capacity. The request-count and cached-token knobs keep their
None-as-unlimited behavior.
Constraint: CP bs>1 batching must not advertise a larger extend batch than chunked prefill can execute.
Rejected: Require users to always set --cp-shared-kv-prefill-max-total-extend-tokens | the safe default is already available from chunked prefill state.
Rejected: Default batch request or cached-token limits | those are policy knobs and None should remain unlimited.
Confidence: high
Scope-risk: narrow
Directive: Keep --cp-shared-kv-prefill-max-total-extend-tokens as min(user_limit, chunk_budget) when both exist.
Tested: Local py_compile for schedule_policy.py and test_prefill_adder.py.
Tested: Remote g0034 cjy-glm5-new targeted prefill_adder tests: 2 passed.
Not-tested: Full ETE scheduler batching distribution after defaulting the extend limit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV batching previously estimated MQA logits from full request
extend/context rows, which overstated memory because CP in-seq split only
computes each rank's two zigzag segments. Add CP-size aware row accounting
that mirrors the fused CP MQA materialization path and take the worst local
rank peak for scheduler admission.
Expose SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_GB as a more direct cap for one fp32
MQA logits chunk. Runtime and scheduler now both translate this GB cap into
chunk rows from the actual K rows, while keeping the old row cap as a
mutually-exclusive expert override.
Constraint: Scheduler admission must stay CUDA-sync-free and use static budget information only.
Rejected: Keep full-request q*k admission | it over-gates CP bs>1 batches because CP splits q rows per rank.
Rejected: Let rows and GB caps both apply | precedence would be ambiguous during tuning.
Confidence: medium
Scope-risk: moderate
Directive: Keep MQA logits admission tied to the fused CP MQA segment shape; do not revert to full request token counts.
Tested: Local py_compile for touched runtime, scheduler, estimator, and tests.
Tested: Local pytest test_cp_shared_kv_prefill_buffer_estimator.py: 8 passed.
Tested: Remote g0034 cjy-glm5-new py_compile and targeted estimator/runtime tests: 13 passed.
Not-tested: Full ETE high-cache-hit CP bs>1 load with SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_GB.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV bs>1 admission already bounds request count, extend tokens,
cached tokens, and an estimated temporary buffer size. The estimate missed
the fp32 MQA logits temporary, whose peak grows with query rows times
context rows and can dominate high-cache-hit multi-request batches.
Add an MQA logits peak term to the CPU-only estimator and include it in
the layer-forward peak enforced by --cp-shared-kv-prefill-max-buffer-size.
When SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_ROWS is set, admission estimates the
post-chunk peak using that row cap; otherwise it remains conservative and
assumes the full extend-row count.
Constraint: Scheduler admission must stay CPU-only and cannot query CUDA free memory.
Rejected: Add a separate scheduler limit for MQA logits | the existing max-buffer-size knob is the right aggregate admission budget.
Rejected: Use SGLANG_NSA_MQA_LOGITS_FREE_MEM_FRACTION in scheduler | that depends on runtime CUDA free memory and would make admission host-sync or stale.
Confidence: medium
Scope-risk: moderate
Directive: Keep the estimator conservative when chunk max rows is unset; do not rely on CUDA free-memory queries in scheduler admission.
Tested: Local py_compile for estimator, scheduler, schedule_policy, and estimator tests.
Tested: Local pytest test_cp_shared_kv_prefill_buffer_estimator.py: 5 passed.
Tested: Remote g0034 cjy-glm5-new py_compile and estimator pytest: 5 passed.
Not-tested: ETE scheduler admission under high-cache-hit bs>1 traffic.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV bs>1 can build large fp32 MQA-logits temporaries from
DeepGEMM fp8_mqa_logits. The official SGLang path already chunks normal NSA
MQA logits by query rows behind a cached memory budget; carry the same budget
control into our NSA indexer and extend it to CP-ragged topk paths that use
row-wise topk_indices_offset_override.
This keeps the previous one-time cached memory-budget behavior rather than the
recent current-free-mem per-forward variant that regressed performance. A new
optional max-rows env provides an explicit hard cap for debugging or controlled
ETE runs without adding host syncs.
Constraint: DeepGEMM materializes fp32 [q, k] logits internally, so row chunking is the narrowest way to cap temporary memory
Rejected: Restore the reverted syh current-free-mem implementation | it changed hot-path heuristics and showed poor runtime performance
Rejected: Split by K/context dimension | would change topk semantics and require a different transform contract
Confidence: medium
Scope-risk: moderate
Directive: CP-ragged chunking relies on topk_indices_offset_override being row-addressed; do not route non-ragged CP paths through it without separate validation
Tested: Local py_compile for environ.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py
Tested: Remote g0034 cjy-glm5-new py_compile for environ.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py
Tested: Remote pytest TestCpSharedKVTaiMaterializeIntegration, 17 passed
Not-tested: CUDA ETE high-cache-hit bs>1 workload memory/performance after chunking
Co-authored-by: OmX <omx@oh-my-codex.dev>
