Reduce CP HiCache capacity synchronization to owner-lane logic
CP shared KV and HiCache now use owner-lane metadata as the authoritative capacity view for host write admission and GPU load-back planning. This removes the debug scalar capacity env and keeps CP load-back from relying on a rank-wide scalar collective when per-owner availability is already known. The load-back planner also accounts for evicting child leaves that unlock ancestor device residency, which fixes small lane deficits despite large aggregate evictable capacity. The commit also adds gated CPU timing logs for CP shared-KV MLA/index prefetch and a CUDA microbenchmark for comparing dense all-reduce with owner-packed all-gather layouts. The timing logs are intentionally behind the existing MLA prefetch log env and should not be enabled for throughput measurements. Constraint: CP shared KV owner lanes require target/draft capacity decisions to preserve page_owners rather than total-token scalars Constraint: CUDA collective benchmarks must run on target GPU hosts, not locally Rejected: Keep SGLANG_CP_HICACHE_CAPACITY_DEBUG observer env | owner-lane admission now replaces that scalar debug path Rejected: Add a silent scalar-allreduce fallback | unexpected owner-lane mismatch should fail fast or log loudly Confidence: medium Scope-risk: moderate Directive: Do not reintroduce CP capacity collectives on the scheduler hot path without proving the owner-lane metadata is insufficient Directive: Disable SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH for end-to-end performance runs; it is diagnostic and high-volume Tested: git diff --check Tested: python -m py_compile on changed runtime/test/benchmark Python files Tested: remote pytest -q test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py (81 passed, 5 warnings) Not-tested: CUDA benchmark benchmark/hicache/bench_cp_shared_kv_prefetch_collective.py Not-tested: full GLM5 E2E throughput after this commit
This commit is contained in:
@@ -185,12 +185,13 @@ waits for D2H write of layer `i`.
|
||||
- Planner skips device-valid nodes, pending backup nodes, host-protected nodes,
|
||||
pinned nodes, malformed metadata, and non-host-backed leaves.
|
||||
- Current eviction behavior is unchanged.
|
||||
- **Online debug option added:** `SGLANG_CP_HICACHE_CAPACITY_DEBUG=1`.
|
||||
- When enabled, write admission computes the deterministic pre-reserve plan and
|
||||
compares its eviction-needed decision with the current collective
|
||||
`reserve_slots_max` result.
|
||||
- Logs use `[HiCache-capacity-debug] write_admission_compare ...`.
|
||||
- This is observer-only and does not remove or add collectives.
|
||||
- **Write admission is now owner-lane authoritative.**
|
||||
- The old observer/debug scalar planner (`current_compatible_need` plus
|
||||
`SGLANG_CP_HICACHE_CAPACITY_DEBUG`) has been removed from the hot path.
|
||||
- Write admission uses the per-owner target/draft deficit vector directly,
|
||||
evicts deterministic host victims only for lanes with deficit, and fails
|
||||
fast if a subsequent local reservation failure contradicts the vector view.
|
||||
- No CP capacity all-reduce is used for write admission.
|
||||
|
||||
### P1: Observer-only ledger
|
||||
|
||||
|
||||
@@ -226,17 +226,12 @@ all_reduce per node victim in a tight host-eviction loop
|
||||
all_reduce every scheduler tick when no completion prefix advanced
|
||||
```
|
||||
|
||||
Current correctness note: CP host reservation now synchronizes
|
||||
`required_host_slots` with `all_reduce(MAX)` before the host-eviction retry.
|
||||
This forces every rank into the same reserve/evict/retry branch and avoids
|
||||
collective mismatches when one rank is host-full and another rank reserves
|
||||
successfully. It is intentionally a coarse slow-path collective, not a
|
||||
per-layer collective, but it can become a performance cost when host pressure
|
||||
is frequent because every reservation failure pays at least one rank-wide MAX
|
||||
sync and retry failures pay a second one. Treat this as a correctness guard to
|
||||
be amortized later with batched reservation epochs, deterministic host
|
||||
watermarks, or less frequent proactive host eviction; do not move it into the
|
||||
per-layer data path.
|
||||
Current correctness note: CP host reservation no longer uses the old
|
||||
`required_host_slots` scalar `all_reduce(MAX)`. The owner-lane vector computed
|
||||
from `CpHiCacheNodeMetadata.page_owners` is now the authoritative admission
|
||||
view for target and draft host capacity. A reservation failure after this
|
||||
vector predicts no deficit is treated as an invariant violation rather than
|
||||
falling back to another collective.
|
||||
|
||||
## Per-Layer Backup Data Plane
|
||||
|
||||
@@ -466,10 +461,11 @@ paths:
|
||||
2. **Pending split behavior.** The chosen first pass is defer/requeue the
|
||||
request that would split a node with pending backup. Do not split the
|
||||
in-flight backup op, including for future `bs > 1`.
|
||||
3. **Ack batching threshold.** Current `writing_check()` can all-reduce on
|
||||
every progress poll. The first per-layer implementation should keep one
|
||||
final logical ack per node and check it at final visibility time, not per
|
||||
layer; batching threshold for final commit remains a later performance pass.
|
||||
3. **Ack batching threshold.** `writing_check()` no longer enters the TP
|
||||
all-reduce while a write is ongoing but no final ack exists. The remaining
|
||||
`MIN` is the final host-visibility barrier for ack entries that do exist:
|
||||
it should not run per layer, and removing it requires a different global
|
||||
visibility protocol rather than a local-only commit.
|
||||
4. **Failure policy under reserve mismatch.** Capacity pressure should skip
|
||||
backup; malformed target/draft metadata should fail fast. If a local rank
|
||||
fails reservation after deterministic host eviction while another succeeds,
|
||||
|
||||
Reference in New Issue
Block a user