Keep CP HiCache draft KV invisible until joint readiness
CP HiCache now treats draft KV as a strict target-owned payload through pending write visibility, host eviction, and state-buffer registration. Host metadata created before async D2H ack is no longer request-visible, so match_prefix cannot select an in-flight host node. Draft host eviction now fails before target cleanup when draft metadata is missing, and prefill/decode share one helper for draft NSA state buffers so shared-KV mode cannot silently skip mismatched draft state. Constraint: CP shared KV + HiCache + EAGLE/MTP must not expose target-only host hits or skipped draft state as valid cache hits Rejected: Rely on event-loop ordering and lock_ref to hide in-flight writes | match_prefix does not consult lock_ref and can observe host_len/cp_hicache directly Rejected: Keep draft state mismatch as debug-only skip | it can poison speculative acceptance while looking like a successful cache hit Confidence: high Scope-risk: moderate Directive: Do not reintroduce silent draft/target fallback in CP shared-KV HiCache paths; malformed strong-sync metadata should fail fast Tested: python -m py_compile targeted modified files Tested: remote g0034 container pytest test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/disaggregation/test_cp_shared_kv_transfer_mapping.py -q (91 passed) Not-tested: Full CP shared KV + HiCache + EAGLE/MTP ETE server run after this commit
This commit is contained in:
@@ -35,6 +35,14 @@ eviction, or visibility decisions.
|
||||
zero-owned rank handling and frees the replay allocation on failure.
|
||||
- Phase 3 implemented: `HiRadixCache.reset()` clears target and draft host pools
|
||||
as one logical reset.
|
||||
- Phase 5 implemented: CP host-backed predicate now treats in-flight
|
||||
write-through nodes as pending, not request-visible host hits; duplicate
|
||||
write-through launch is suppressed while a node is in
|
||||
`ongoing_write_through`.
|
||||
- Phase 6 implemented: controller tests now cover target+draft host eviction,
|
||||
zero-owned draft metadata, and draft-before-target load readiness; CP draft
|
||||
state-buffer registration is centralized and fails fast on target/draft state
|
||||
mismatch when `SGLANG_CP_DRAFT_SHARED_KV=1`.
|
||||
- Adjacent controller invariant guard restored for CPU/fake-test tensors only:
|
||||
page-shaped host/device indices are validated in unit tests without
|
||||
reintroducing CUDA tensor truth-value checks or hot-path host sync.
|
||||
@@ -574,6 +582,14 @@ the explicit-ready path before per-layer backup. Pending write is a valid
|
||||
lifecycle state, so it may return “not ready”; malformed metadata must still
|
||||
raise.
|
||||
|
||||
Implemented decision: use the explicit-ready path. `_node_backuped()` still
|
||||
validates CP/draft metadata first, but returns `False` while the node id is in
|
||||
`ongoing_write_through`. This keeps in-flight D2H metadata out of request-visible
|
||||
host-hit admission until `writing_check()` observes the joint target+draft ack
|
||||
and removes the node from the pending table. `_inc_hit_count()` treats pending
|
||||
write-through as “already launched” and does not start a second write for the
|
||||
same node.
|
||||
|
||||
### Phase 6: Keep internal transfer split, but expose one logical operation
|
||||
|
||||
Files:
|
||||
@@ -615,6 +631,23 @@ Add controller tests:
|
||||
an error. It should not silently skip draft state buffers because that creates
|
||||
a target/draft cache-hit mismatch.
|
||||
|
||||
Implemented status:
|
||||
|
||||
- `evict_cp_host()` fails before freeing target host slots if draft HiCache is
|
||||
attached but metadata lacks `draft_host_indices`; otherwise target and draft
|
||||
host slots are freed as one logical operation.
|
||||
- `_write_cp()` zero-owned ranks keep explicit empty `draft_host_indices`, so
|
||||
zero-owned target metadata is still draft-valid.
|
||||
- `start_loading()` is test-protected to submit draft H2D before completing the
|
||||
corresponding target layer readiness event.
|
||||
- `append_cp_draft_state_buffers()` is the single helper used by prefill and
|
||||
decode queue setup. It appends draft NSA state buffers to the target state
|
||||
stream, preserves legacy skip behavior when draft shared KV is disabled, and
|
||||
raises on target/draft state mismatch when draft shared KV is enabled.
|
||||
- Code-read confirmation: PD transfer page indices remain target-driven;
|
||||
draft KV buffers are extra buffers behind the same page-index stream, not an
|
||||
independent draft page-index stream.
|
||||
|
||||
### Phase 7: Remote validation
|
||||
|
||||
Run targeted tests in the remote container because this repo's runtime deps and
|
||||
|
||||
Reference in New Issue
Block a user