Revert the tail-chunk co-batching gate from 54c056af because allowing a continued chunk to share the next CP bs>1 batch reopens the mixed chunk/page-tail scheduler risks we are currently avoiding. Keep the independent real-prefix budget accounting so chunked requests still contribute their carried prefix to CP cached-token and buffer estimates.\n\nConstraint: Chunked-prefill requests must remain solo until the CP split/page-tail contract is revalidated for mixed batches.\nRejected: Full revert of 54c056af | it would also drop true-prefix budget accounting and under-estimate cache/buffer pressure for admitted chunks.\nConfidence: high\nScope-risk: moderate\nDirective: Do not reintroduce tail-chunk co-batching without tests covering page-tail split, CP buffer admission, and ETE chunked+cache-hit replay.\nTested: Local py_compile for environ.py, schedule_policy.py, cp_shared_kv_compose.py, test_prefill_adder.py, test_cp_shared_kv_compose_v2_8rank.py.\nTested: Remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/managers/test_prefill_adder.py -> 25 passed.\nTested: Remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 157 passed, 2 subtests passed.\nNot-tested: Full mixed replay with chunked-prefill traffic after service restart.
The bench gains a pipelined mode (topk_N event-gated on a side stream,
overlapping logits_{N+1}). Byte-equality validation was dropped after
establishing the kernel chain is run-to-run nondeterministic even
serial-vs-serial (near-equal fp32 selection).
Result on g0033 H200: pipelining recovers only 0.3-8.9% where small
chunks cost +15-46% — fp8_mqa_logits and fast_topk_transform_fused are
both SM-saturating, so concurrent streams timeshare instead of
overlapping; the small-chunk penalty is small-M GEMM inefficiency. No
production pipeline path; the serial loop at CHUNK_MAX_GB=2 stands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
专题 S1 (design: docs_internal/perf/prefill-compute-intensity-plan.md
S1.0-S1.11). A batch containing a chunked prefill request has been
forced to bs=1 by the CP gate, so every chunk of a long prompt
monopolizes a forward while short-extend cache-hit continuations queue
— the direct cause of the replay TTFT tail (p90 19.3s / p99 49.2s at
91.8% cache hit). Yet mixed chunk batches already occur today (a
freshly-chunked request keeps earlier-admitted small requests), proving
the CP forward path is mixed-chunk-safe; only admission was asymmetric.
Three changes, the first flag-independent:
- add_chunked_req now seeds the budget with the chunk's TRUE prefix
(was 0), so the CP cached tally and the buffer estimator's mqa_logits
k_rows see the chunk's footprint before any later request is admitted
(landmine D1).
- New SGLANG_CP_PREFILL_MIX_CHUNKED (default OFF): with it on, the gate
admits requests after a chunked one and lets the existing CP caps
(extend / cached / buffer, now correctly seeded) bound the batch — a
FULL chunk still ends the scan by consuming the chunk-clamped extend
cap; only a tail chunk leaves headroom. A chunked prefix that is not
page-aligned (rare sub-page final-chunk tail) keeps its batch solo
(the CP page-aligned split would fail-fast otherwise).
- The symm staging capacity identity (admission extend cap + request
slack == staging pages) is asserted when the flag is on, locking the
coupling the design relies on (plan doc S1.4 I2).
Tests: 4 new adder units (budget seeding; tail chunk admits followers;
full chunk solo by budget; non-aligned prefix solo); the 8-rank
byte-exactness scenario gains a chunk-shaped request (2048-token
page-aligned carried prefix + 512 extend) — all four phases
(legacy/v2/symm/prefetch) byte-identical on g0033. Known pre-existing
cross-file pollution noted in problems.md P16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measures the real cost of shrinking SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_GB:
the faithful indexer loop (deep_gemm.fp8_mqa_logits +
fast_topk_transform_fused, serial) at GLM-5.1 shapes (H=32, D=128,
topk=2048) across cold-chunk / tail-chunk / warm-continuation /
warm-long scenarios. g0033 1xH200 results: 2GB costs at most +5.5%
(cold 64K chunk) and is -6.7% on the heaviest warm-long shape; the
knee is ~1GB; 0.5GB is +30%. Shrinking 8->2GB frees ~6GB of the
per-batch CP admission budget for KV layer buffers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First e2e launch crashed every CP rank at the symm token fill:
index_copy_cuda is not implemented for Float8_e4m3fn — the production
KV pool dtype, which the 8-rank test missed by building its pools as
uint8. The fill is a whole-token-row copy, so it is dtype-agnostic:
both the staging span and the current rows now go through uint8 views.
The 8-rank test now builds the KV pools and current rows as
float8_e4m3fn (payloads constructed as bytes, compared as bytes) so
the production dtype is what every phase exercises.
Validated on g0033: all four 8-rank byte-exactness phases under fp8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bs=1 MLA/index prefetchers replaced their consume-side trailing-range
NCCL all-reduce with the staging exchange: fill current rows straight into
this round's staging span (token-KV collapses to one cached index_copy;
the index fill kernel is just pointed at the staging page inverse),
cp_symm_barrier, then gather ALL current pages — this rank's own included
— from the stagings into the prefetched dense buffer. The symm+prefetcher
FAIL_FAST is gone.
Rank-uniformity moves with it: staging registration now also happens in
maybe_create (batch-logical gates, before any per-rank miss can diverge),
because with a prefetcher active the sync compose runs only on per-rank
misses and its lazy collective registration would hang. A hit/miss
divergence itself stays barrier-safe — both the prefetch consume and the
sync-compose fallback execute exactly one begin_round + barrier per
(layer, kind), and the counting barrier is shape-free (unlike the AR pair
it replaces, which would shape-mismatch).
Found by the new index test phase: the fill/remap kernel family skips
page id 0 as the SGLang dummy page, so a 0-based first staging slot was
never written. The staging layout now reserves row 0 (slot of current
page i = i + 1) for every kind, matching the convention instead of
depending on per-kernel behavior.
Launch-path cost: per-(kind,parity) peer pointer tables and the
[pool|staging] concatenations are precomputed/cached (identity pinned by
holding the pool-table reference); all prefetch descriptors, staging row
indices, and mixed_locs are built once per batch.
Validated on g0033 8xH200: 151 unit tests; 8-rank byte-exactness for
token sync symm (8 layers), index sync symm (4 layers, new phase), and
MLA + index prefetch consume_prefix_with_current vs the legacy sync
compose.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The publish-variant staging exchange measured ~equal to the default
compact-current AR (90.4 vs 86.5 ms/batch on the traced scenario): the
publish copy and the barrier serialized behind the 0.65 ms prefix
gather ate the transport win the isolated current exchange showed
(0.196 vs 0.354 ms). Fix the structure instead of the copy: current
rows are now written straight INTO the staging — the fill kernels take
their write destinations solely from page_inverse, so a per-batch
staging-remapped page inverse on the plan retargets them with zero
kernel changes — then cp_symm_barrier, then ONE slot-dense gather
covers prefix pages (pool pointers) and ALL current pages (staging
pointers, including this rank's own) through a concatenated 2*cp
pointer table where current slots carry owner = cp_size + writer and
src = staging slot. No publish copy, no prefix pre-gather, no second
gather.
The fused fill's loc outputs are dense-geometry-bound, so the token-KV
path computes mixed_locs/staging row indices once per batch (they are
layer-invariant) and the per-layer fill collapses to a single
index_copy_ into the zeroed staging span.
Benchmark (g0033 8xH200, byte-exact, idle-checked): 62.8 ms/batch vs
84.4 default Step A (-26%) and 60.8 ideal; publish variant was 88.0.
151 unit tests; 8-rank GPU byte-exactness vs v2 across 8 layers,
arena on and off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Peers only ever read the CURRENT pages of a rank's compose output — the
prefix comes straight from the IPC-registered KV pool — so the symm
region does not need to hold the whole dense buffer (pool-bound ~2.5 GB
double-buffered slab). It now holds one round of current pages in
merged-span order (extend-cap-bound, ~58-100 MB), and dense buffers
become purely rank-local (plain allocations or the optional local arena;
COMPOSE_SYMM no longer requires COMPOSE_ARENA).
Exchange per compose call: publish my written current pages
dense[page] -> staging[slot i] (slot = the page's batch current index,
identical on every rank, so peers address each other's staging with no
per-batch handshake), cp_symm_barrier, gather peers'
staging[writer][slot] -> dense[page] via the existing src!=dst page
gather. Reuse safety keeps the parity-half distance-2 argument, now on
the staging. Capacity sizing comes from the admission caps
(max_total_extend_tokens / max_batch_requests) with a pool-derived
fallback and the SYMM_HEAP_MB override; overflow fails fast
(batch-logical, hence rank-uniform).
Idea credit: laoyao0822's touched-pages-proportional staging
(906ecbe5d4), rebound onto our barrier-gated, group-agreed transport.
Validated on g0033 8xH200: 151 unit tests; 8-rank GPU byte-exactness
vs compose_v2 across 8 layers (arena on and off, parity halves
exercised); benchmark path e (real protocol) byte-exact, current-page
exchange 0.196 ms vs 0.354 ms compact-AR isolated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From the nsys CPU-gap attribution (launch thread, one 78-layer forward:
374ms API time; 642 cudaStreamSynchronize blocking 89.5ms and overlapping
122ms of the 505ms GPU idle; ~44ms pure-Python before concat_mla_absorb_q):
- memory_pool_host: skip validate_page_aligned_token_indices on CUDA
tensors in _get_indexer_page_indices and _prepare_load_page_indices —
torch.any/torch.equal there cost a queue-deep cudaStreamSynchronize per
layer-group submit (~0.42ms each, ~12.7ms/forward measured). Same
construction-based-invariant guard the CacheController pair check
already documents; CPU/test tensors stay validated.
- nsa_indexer: per-batch _CpRaggedIndexPlan replaces the per-F-layer
rebuild of the O(total-q-tokens) topk offset list and the 6-7 int32
ragged descriptor tensors (segment records, kv_lens/q_starts/q_lens/
k_bases/q_bases/current_bases). All inputs are batch metadata; the plan
is anchored on the forward batch with a content key over cp_index.
- nsa_indexer forward_indexer: read seq_lens_cpu instead of a device
seq_lens[i].item() per request per layer (one stream sync each).
- cp_shared_kv_runtime: get_or_build_batch_slot_spans caches the
layer-invariant prefix/current slot spans per batch (the builders read
logical_pages only for its shape); nsa_backend x3 + nsa_indexer call
sites switched.
Microbenchmark (idle H200, traced batch shape bs=12 / 44.6K q tokens,
test/manual/bench_cpu_gap_fixes.py, equality-checked): validator path
197.1us -> 59.2us per submit under a busy queue (x3.3); ragged plan
3238.6us -> 36.2us per layer (x90, ~128ms launch-thread time per forward
at 40 F-layers); slot spans 20.1us -> 0.5us (x41). Layer suites A/B vs
HEAD: identical failure set (5 pre-existing CPU-tensor indexer tests),
no regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the last remaining NCCL collective in the bs>1 compose layer loop
(the compact-current all-reduce) with a DeepEP-style counting barrier plus
a CUDA-IPC gather from peers' symmetric dense buffers, behind
SGLANG_CP_SHARED_KV_COMPOSE_SYMM (+ COMPOSE_ARENA, both default off).
- CpComposeArena.register_symm: fixes capacity at the pool-derived bound
(logical pages x dense page unit, overridable via
SGLANG_CP_SHARED_KV_SYMM_HEAP_MB), allocates slab + flags in CUDA-IPC
memory, exchanges handles once over the CP group; deterministic bump
carve means peer_base + my_offset addresses any peer's dense buffer with
no per-layer handshakes. Registration happens only in the token-KV
compose (uniform first-use point); growth after registration raises.
- Current pages are single-writer at page granularity under the
page-aligned in-seq split, so the exchange is the existing
gather_cuda_ipc_peer_pages with src==dst page ids and writer (compute
owner) ranks; writers are built per batch by
build_batch_current_page_writer_ranks and gated by
maybe_build_current_page_writer_ranks (page_aligned metadata required).
The barrier runs even with zero remote pages (counts must match).
- _agreed_tai_ipc_peer_ptrs: the per-rank IPC capability probe is now
agreed across the CP group (one-time MIN all-reduce per pool tensor) so
ranks can never split between gather and collective paths and deadlock
on mismatched NCCL shapes.
- ComposePlan cache re-anchored ON the slot_remap object (forward-batch
lifetime) instead of a module-level tensor-identity key, which could go
stale when a freed tensor's address is reused by the next batch.
Validation (g0034 syh-dev-new): tai-kernel cp_symm_barrier correctness
(200 adversarial iterations, rotating 10ms producer delays, phase-safety,
flags drained at quiescence) and perf (7.7us max-rank latency) both pass;
8-rank GPU test extends to the symm path - byte-identical to compose_v2
across 4 layers (parity halves exercised, slab registered); mem_cache
suite 464 passed with the only failures being a documented pre-existing
sys.modules stub pollution pair, reproduced identically with
SGLANG_CP_SHARED_KV_COMPOSE_V2=0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bs>1 partial-current materialize issued one sum-all-reduce per request
span per buffer per layer (48 collectives per F-layer at bs=12, 2880 per
batch, 419ms + 49ms launch gaps in the production trace), all inline on
the compute stream. The data is a partitioned gather, not a reduction:
every byte has exactly one producer.
compose_v2 (SGLANG_CP_SHARED_KV_COMPOSE_V2, default on) replaces this with:
- fast path: one tai-kernel CUDA-IPC slot-dense gather covering ALL prefix
spans (full-range descriptors, -1 sentinels zero-fill current slots and
replace the dense zero-fill) + ONE collective over the compact current
pages (uint8 byte view; exact because every byte is writer-exclusive).
- fallback (no peer IPC): local materialize of all prefix spans + ONE
whole-buffer sum-all-reduce (rows are still writer-exclusive pre-reduce).
IPC capability is decided once by the cached peer-pointer probe; after a
successful probe a failing gather raises (no per-call try/except).
cp_shared_kv_compose.py adds the per-batch ComposePlan descriptor cache
(layer-invariant, keyed on the slot_logical_pages identity) and the
CpComposeArena with tier-S carve discipline (deterministic bump, layer-
parity halves; default off) so the Step B symmetric-memory conversion is
a registration flip.
Microbenchmark (g0034 8xH200, traced 12-req batch, per batch): per-span
214ms -> fused AR 119ms -> IPC prefix + compact current 84ms; symm target
61ms. Validation: 143 unit tests incl. v2-contract twins, legacy siblings
and rank-merged simulations under both paths; mem_cache dir 432 passed;
8-rank GPU byte-exactness vs legacy with real NCCL + IPC (test/manual/
test_cp_shared_kv_compose_v2_8rank.py) passed with no fallback markers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
