Stabilize CP shared-KV batch padding semantics

CP shared-KV bs>1 exposed three distinct padding domains: valid cache rows, CP page-tail compute rows, and MLP-sync flattened static padding. The previous implementation mixed these domains in direct-write and index top-k paths, so real requests failed when q/out_cache_loc lengths matched valid rows while metadata aliases described compute rows.\n\nThis change makes compute split strip only proven flattened static padding, keeps valid cache writes strict except for extend_num_tokens-proven static tails, marks CP-local EAGLE draft hidden state explicitly, and selects NSA index top-k query metadata by the actual q/weight row count.\n\nConstraint: CP shared-KV cache writes must never persist dummy page-tail or MLP static padding rows.\nConstraint: EAGLE draft hidden state can be CP-local before full CP metadata is visible in prepare_mlp_sync_batch.\nRejected: Use compute_padding_enabled as direct-write truncation proof | it silently accepts unknown out_cache_loc tails.\nRejected: Always consume compute q metadata in index top-k | actual q/weights can be valid-only after CP split.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not collapse valid rows, CP compute padding, and MLP static padding into one length condition; use explicit provenance.\nTested: remote py_compile for touched NSA files\nTested: remote targeted CP shared-KV padding/top-k regressions\nTested: remote pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py -k 'not test_tai_current_slot_fill_sparse_page_self_test_passes_on_installed_kernel' => 228 passed, 1 deselected, 5 warnings, 2 subtests passed\nNot-tested: full ETE replay after the final index top-k fix\nNot-tested: TAI current-index fast path dtype fallback
This commit is contained in:
laoyao0822
2026-06-04 07:25:11 +08:00
parent 02af370e87
commit 3d6007246b
7 changed files with 1403 additions and 60 deletions
@@ -622,6 +622,10 @@ class EagleDraftInput(SpecInput, EagleDraftInputV2Mixin):
topk_index: torch.Tensor = None
# shape: (b, hidden_size)
hidden_states: torch.Tensor = None
# True when hidden_states is a CP-local side channel captured by the target
# model before CP output collect. This is a semantic marker; consumers must
# not infer it from tensor length alone.
cp_local_hidden_states: bool = False
capture_hidden_mode: CaptureHiddenMode = CaptureHiddenMode.FULL
# Inputs for extend
@@ -334,6 +334,9 @@ class EAGLEWorker(TpModelWorker):
if logits_output.draft_hidden_states is not None
else logits_output.hidden_states
)
cp_local_draft_hidden_states = (
logits_output.draft_hidden_states is not None
)
if (
envs.SGLANG_CP_DRAFT_SHARED_KV.get()
and draft_hidden_states is None
@@ -347,6 +350,7 @@ class EAGLEWorker(TpModelWorker):
next_token_ids,
seq_lens_cpu,
logits_output.mm_input_embeds,
cp_local_hidden_states=cp_local_draft_hidden_states,
)
return GenerationBatchResult(
logits_output=logits_output,
@@ -935,6 +939,8 @@ class EAGLEWorker(TpModelWorker):
next_token_ids: torch.Tensor,
seq_lens_cpu: Optional[torch.Tensor],
mm_input_embeds: Optional[torch.Tensor] = None,
*,
cp_local_hidden_states: bool = False,
):
"""Run draft model extend. This API modifies the states of the batch.
@@ -945,6 +951,7 @@ class EAGLEWorker(TpModelWorker):
"""
batch.spec_info = EagleDraftInput(
hidden_states=hidden_states,
cp_local_hidden_states=cp_local_hidden_states,
verified_id=next_token_ids,
num_tokens_per_req=1,
num_tokens_for_logprob_per_req=1,