dbc5ebbaa0968f0fe6ef4e86d5ad8ea61770cce2
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  leaveletandClaude Opus 4.8
|
e23168e7f5 |
Fix CP shared-KV bs=1 cache-hit zero-prefix corruption; spans are sole compose input
A bs=1 cache-hit prefill produced deterministically wrong output (e.g. "0.5,0.5,0.5").
Root cause: regression
|
||
|
laoyao0822
|
ee843a946b |
Keep chunked CP prefills solo during bs>1 admission
Revert the tail-chunk co-batching gate from |
||
|
leaveletandClaude Fable 5
|
54c056af83 |
Let tail chunks co-batch with cache-hit requests (SGLANG_CP_PREFILL_MIX_CHUNKED)
专题 S1 (design: docs_internal/perf/prefill-compute-intensity-plan.md S1.0-S1.11). A batch containing a chunked prefill request has been forced to bs=1 by the CP gate, so every chunk of a long prompt monopolizes a forward while short-extend cache-hit continuations queue — the direct cause of the replay TTFT tail (p90 19.3s / p99 49.2s at 91.8% cache hit). Yet mixed chunk batches already occur today (a freshly-chunked request keeps earlier-admitted small requests), proving the CP forward path is mixed-chunk-safe; only admission was asymmetric. Three changes, the first flag-independent: - add_chunked_req now seeds the budget with the chunk's TRUE prefix (was 0), so the CP cached tally and the buffer estimator's mqa_logits k_rows see the chunk's footprint before any later request is admitted (landmine D1). - New SGLANG_CP_PREFILL_MIX_CHUNKED (default OFF): with it on, the gate admits requests after a chunked one and lets the existing CP caps (extend / cached / buffer, now correctly seeded) bound the batch — a FULL chunk still ends the scan by consuming the chunk-clamped extend cap; only a tail chunk leaves headroom. A chunked prefix that is not page-aligned (rare sub-page final-chunk tail) keeps its batch solo (the CP page-aligned split would fail-fast otherwise). - The symm staging capacity identity (admission extend cap + request slack == staging pages) is asserted when the flag is on, locking the coupling the design relies on (plan doc S1.4 I2). Tests: 4 new adder units (budget seeding; tail chunk admits followers; full chunk solo by budget; non-aligned prefix solo); the 8-rank byte-exactness scenario gains a chunk-shaped request (2048-token page-aligned carried prefix + 512 extend) — all four phases (legacy/v2/symm/prefetch) byte-identical on g0033. Known pre-existing cross-file pollution noted in problems.md P16. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
leaveletandClaude Fable 5
|
d66758d929 |
Run the symm staging fill on byte views (no fp8 index_copy kernel)
First e2e launch crashed every CP rank at the symm token fill: index_copy_cuda is not implemented for Float8_e4m3fn — the production KV pool dtype, which the 8-rank test missed by building its pools as uint8. The fill is a whole-token-row copy, so it is dtype-agnostic: both the staging span and the current rows now go through uint8 views. The 8-rank test now builds the KV pools and current rows as float8_e4m3fn (payloads constructed as bytes, compared as bytes) so the production dtype is what every phase exercises. Validated on g0033: all four 8-rank byte-exactness phases under fp8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
leaveletandClaude Fable 5
|
1923fcd67e |
Wire the symm current exchange into the CP shared-KV prefetchers
The bs=1 MLA/index prefetchers replaced their consume-side trailing-range NCCL all-reduce with the staging exchange: fill current rows straight into this round's staging span (token-KV collapses to one cached index_copy; the index fill kernel is just pointed at the staging page inverse), cp_symm_barrier, then gather ALL current pages — this rank's own included — from the stagings into the prefetched dense buffer. The symm+prefetcher FAIL_FAST is gone. Rank-uniformity moves with it: staging registration now also happens in maybe_create (batch-logical gates, before any per-rank miss can diverge), because with a prefetcher active the sync compose runs only on per-rank misses and its lazy collective registration would hang. A hit/miss divergence itself stays barrier-safe — both the prefetch consume and the sync-compose fallback execute exactly one begin_round + barrier per (layer, kind), and the counting barrier is shape-free (unlike the AR pair it replaces, which would shape-mismatch). Found by the new index test phase: the fill/remap kernel family skips page id 0 as the SGLang dummy page, so a 0-based first staging slot was never written. The staging layout now reserves row 0 (slot of current page i = i + 1) for every kind, matching the convention instead of depending on per-kernel behavior. Launch-path cost: per-(kind,parity) peer pointer tables and the [pool|staging] concatenations are precomputed/cached (identity pinned by holding the pool-table reference); all prefetch descriptors, staging row indices, and mixed_locs are built once per batch. Validated on g0033 8xH200: 151 unit tests; 8-rank byte-exactness for token sync symm (8 layers), index sync symm (4 layers, new phase), and MLA + index prefetch consume_prefix_with_current vs the legacy sync compose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
leaveletandClaude Fable 5
|
d63fbd4d79 |
Fold the symm compose into one barrier-gated mega gather
The publish-variant staging exchange measured ~equal to the default compact-current AR (90.4 vs 86.5 ms/batch on the traced scenario): the publish copy and the barrier serialized behind the 0.65 ms prefix gather ate the transport win the isolated current exchange showed (0.196 vs 0.354 ms). Fix the structure instead of the copy: current rows are now written straight INTO the staging — the fill kernels take their write destinations solely from page_inverse, so a per-batch staging-remapped page inverse on the plan retargets them with zero kernel changes — then cp_symm_barrier, then ONE slot-dense gather covers prefix pages (pool pointers) and ALL current pages (staging pointers, including this rank's own) through a concatenated 2*cp pointer table where current slots carry owner = cp_size + writer and src = staging slot. No publish copy, no prefix pre-gather, no second gather. The fused fill's loc outputs are dense-geometry-bound, so the token-KV path computes mixed_locs/staging row indices once per batch (they are layer-invariant) and the per-layer fill collapses to a single index_copy_ into the zeroed staging span. Benchmark (g0033 8xH200, byte-exact, idle-checked): 62.8 ms/batch vs 84.4 default Step A (-26%) and 60.8 ideal; publish variant was 88.0. 151 unit tests; 8-rank GPU byte-exactness vs v2 across 8 layers, arena on and off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
leaveletandClaude Fable 5
|
e8e5edb230 |
Shrink the symm compose region to a compact current-page staging
Peers only ever read the CURRENT pages of a rank's compose output — the prefix comes straight from the IPC-registered KV pool — so the symm region does not need to hold the whole dense buffer (pool-bound ~2.5 GB double-buffered slab). It now holds one round of current pages in merged-span order (extend-cap-bound, ~58-100 MB), and dense buffers become purely rank-local (plain allocations or the optional local arena; COMPOSE_SYMM no longer requires COMPOSE_ARENA). Exchange per compose call: publish my written current pages dense[page] -> staging[slot i] (slot = the page's batch current index, identical on every rank, so peers address each other's staging with no per-batch handshake), cp_symm_barrier, gather peers' staging[writer][slot] -> dense[page] via the existing src!=dst page gather. Reuse safety keeps the parity-half distance-2 argument, now on the staging. Capacity sizing comes from the admission caps (max_total_extend_tokens / max_batch_requests) with a pool-derived fallback and the SYMM_HEAP_MB override; overflow fails fast (batch-logical, hence rank-uniform). Idea credit: laoyao0822's touched-pages-proportional staging (906ecbe5d4), rebound onto our barrier-gated, group-agreed transport. Validated on g0033 8xH200: 151 unit tests; 8-rank GPU byte-exactness vs compose_v2 across 8 layers (arena on and off, parity halves exercised); benchmark path e (real protocol) byte-exact, current-page exchange 0.196 ms vs 0.354 ms compact-AR isolated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
leaveletandClaude Fable 5
|
c55e406176 |
Add symm-heap current-page exchange for CP shared-KV compose (Step B)
Replaces the last remaining NCCL collective in the bs>1 compose layer loop (the compact-current all-reduce) with a DeepEP-style counting barrier plus a CUDA-IPC gather from peers' symmetric dense buffers, behind SGLANG_CP_SHARED_KV_COMPOSE_SYMM (+ COMPOSE_ARENA, both default off). - CpComposeArena.register_symm: fixes capacity at the pool-derived bound (logical pages x dense page unit, overridable via SGLANG_CP_SHARED_KV_SYMM_HEAP_MB), allocates slab + flags in CUDA-IPC memory, exchanges handles once over the CP group; deterministic bump carve means peer_base + my_offset addresses any peer's dense buffer with no per-layer handshakes. Registration happens only in the token-KV compose (uniform first-use point); growth after registration raises. - Current pages are single-writer at page granularity under the page-aligned in-seq split, so the exchange is the existing gather_cuda_ipc_peer_pages with src==dst page ids and writer (compute owner) ranks; writers are built per batch by build_batch_current_page_writer_ranks and gated by maybe_build_current_page_writer_ranks (page_aligned metadata required). The barrier runs even with zero remote pages (counts must match). - _agreed_tai_ipc_peer_ptrs: the per-rank IPC capability probe is now agreed across the CP group (one-time MIN all-reduce per pool tensor) so ranks can never split between gather and collective paths and deadlock on mismatched NCCL shapes. - ComposePlan cache re-anchored ON the slot_remap object (forward-batch lifetime) instead of a module-level tensor-identity key, which could go stale when a freed tensor's address is reused by the next batch. Validation (g0034 syh-dev-new): tai-kernel cp_symm_barrier correctness (200 adversarial iterations, rotating 10ms producer delays, phase-safety, flags drained at quiescence) and perf (7.7us max-rank latency) both pass; 8-rank GPU test extends to the symm path - byte-identical to compose_v2 across 4 layers (parity halves exercised, slab registered); mem_cache suite 464 passed with the only failures being a documented pre-existing sys.modules stub pollution pair, reproduced identically with SGLANG_CP_SHARED_KV_COMPOSE_V2=0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
leaveletandClaude Fable 5
|
f47b739e30 |
Restructure bs>1 CP shared-KV compose to one gather + one collective
The bs>1 partial-current materialize issued one sum-all-reduce per request span per buffer per layer (48 collectives per F-layer at bs=12, 2880 per batch, 419ms + 49ms launch gaps in the production trace), all inline on the compute stream. The data is a partitioned gather, not a reduction: every byte has exactly one producer. compose_v2 (SGLANG_CP_SHARED_KV_COMPOSE_V2, default on) replaces this with: - fast path: one tai-kernel CUDA-IPC slot-dense gather covering ALL prefix spans (full-range descriptors, -1 sentinels zero-fill current slots and replace the dense zero-fill) + ONE collective over the compact current pages (uint8 byte view; exact because every byte is writer-exclusive). - fallback (no peer IPC): local materialize of all prefix spans + ONE whole-buffer sum-all-reduce (rows are still writer-exclusive pre-reduce). IPC capability is decided once by the cached peer-pointer probe; after a successful probe a failing gather raises (no per-call try/except). cp_shared_kv_compose.py adds the per-batch ComposePlan descriptor cache (layer-invariant, keyed on the slot_logical_pages identity) and the CpComposeArena with tier-S carve discipline (deterministic bump, layer- parity halves; default off) so the Step B symmetric-memory conversion is a registration flip. Microbenchmark (g0034 8xH200, traced 12-req batch, per batch): per-span 214ms -> fused AR 119ms -> IPC prefix + compact current 84ms; symm target 61ms. Validation: 143 unit tests incl. v2-contract twins, legacy siblings and rank-merged simulations under both paths; mem_cache dir 432 passed; 8-rank GPU byte-exactness vs legacy with real NCCL + IPC (test/manual/ test_cp_shared_kv_compose_v2_8rank.py) passed with no fallback markers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |