Stabilize CP shared-KV batch padding semantics
CP shared-KV bs>1 exposed three distinct padding domains: valid cache rows, CP page-tail compute rows, and MLP-sync flattened static padding. The previous implementation mixed these domains in direct-write and index top-k paths, so real requests failed when q/out_cache_loc lengths matched valid rows while metadata aliases described compute rows.\n\nThis change makes compute split strip only proven flattened static padding, keeps valid cache writes strict except for extend_num_tokens-proven static tails, marks CP-local EAGLE draft hidden state explicitly, and selects NSA index top-k query metadata by the actual q/weight row count.\n\nConstraint: CP shared-KV cache writes must never persist dummy page-tail or MLP static padding rows.\nConstraint: EAGLE draft hidden state can be CP-local before full CP metadata is visible in prepare_mlp_sync_batch.\nRejected: Use compute_padding_enabled as direct-write truncation proof | it silently accepts unknown out_cache_loc tails.\nRejected: Always consume compute q metadata in index top-k | actual q/weights can be valid-only after CP split.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not collapse valid rows, CP compute padding, and MLP static padding into one length condition; use explicit provenance.\nTested: remote py_compile for touched NSA files\nTested: remote targeted CP shared-KV padding/top-k regressions\nTested: remote pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py -k 'not test_tai_current_slot_fill_sparse_page_self_test_passes_on_installed_kernel' => 228 passed, 1 deselected, 5 warnings, 2 subtests passed\nNot-tested: full ETE replay after the final index top-k fix\nNot-tested: TAI current-index fast path dtype fallback
This commit is contained in:
@@ -622,6 +622,10 @@ class EagleDraftInput(SpecInput, EagleDraftInputV2Mixin):
|
||||
topk_index: torch.Tensor = None
|
||||
# shape: (b, hidden_size)
|
||||
hidden_states: torch.Tensor = None
|
||||
# True when hidden_states is a CP-local side channel captured by the target
|
||||
# model before CP output collect. This is a semantic marker; consumers must
|
||||
# not infer it from tensor length alone.
|
||||
cp_local_hidden_states: bool = False
|
||||
capture_hidden_mode: CaptureHiddenMode = CaptureHiddenMode.FULL
|
||||
|
||||
# Inputs for extend
|
||||
|
||||
@@ -334,6 +334,9 @@ class EAGLEWorker(TpModelWorker):
|
||||
if logits_output.draft_hidden_states is not None
|
||||
else logits_output.hidden_states
|
||||
)
|
||||
cp_local_draft_hidden_states = (
|
||||
logits_output.draft_hidden_states is not None
|
||||
)
|
||||
if (
|
||||
envs.SGLANG_CP_DRAFT_SHARED_KV.get()
|
||||
and draft_hidden_states is None
|
||||
@@ -347,6 +350,7 @@ class EAGLEWorker(TpModelWorker):
|
||||
next_token_ids,
|
||||
seq_lens_cpu,
|
||||
logits_output.mm_input_embeds,
|
||||
cp_local_hidden_states=cp_local_draft_hidden_states,
|
||||
)
|
||||
return GenerationBatchResult(
|
||||
logits_output=logits_output,
|
||||
@@ -935,6 +939,8 @@ class EAGLEWorker(TpModelWorker):
|
||||
next_token_ids: torch.Tensor,
|
||||
seq_lens_cpu: Optional[torch.Tensor],
|
||||
mm_input_embeds: Optional[torch.Tensor] = None,
|
||||
*,
|
||||
cp_local_hidden_states: bool = False,
|
||||
):
|
||||
"""Run draft model extend. This API modifies the states of the batch.
|
||||
|
||||
@@ -945,6 +951,7 @@ class EAGLEWorker(TpModelWorker):
|
||||
"""
|
||||
batch.spec_info = EagleDraftInput(
|
||||
hidden_states=hidden_states,
|
||||
cp_local_hidden_states=cp_local_hidden_states,
|
||||
verified_id=next_token_ids,
|
||||
num_tokens_per_req=1,
|
||||
num_tokens_for_logprob_per_req=1,
|
||||
|
||||
Reference in New Issue
Block a user