Prevent batched CP draft from silently leaving the target path
EAGLE draft shared-KV is supposed to mirror the target CP layout, so bs>1 must not fall back to legacy full-input or padded-hidden behavior when required batch metadata is missing or inconsistent. This change keeps the existing bs=1 compatibility path but makes batched CP draft fail fast on missing/mismatched spec hidden states, embedding pad metadata, or input embed shapes. The docs record the current W7 boundary: draft prefill follows target metadata, while scheduler admission and ETE remain gated. Constraint: CP draft KV must mirror target layout and must not silently diverge under bs>1 shared-KV. Rejected: Allow bs>1 to use the old full-input fallback | it can hide wrong owner/page metadata and corrupt accept length. Confidence: medium Scope-risk: moderate Directive: Do not open the scheduler bs>1 CP gate until EAGLE accept length/output length are verified with this fail-fast path enabled. Tested: Remote g0034 targeted EAGLE fail-fast unit test passed; remote full test/registered/unit/layers/test_nsa_cp_utils.py passed 70 tests. Not-tested: EAGLE bs>1 ETE, because scheduler CP bs>1 admission gate remains closed.
This commit is contained in:
@@ -557,6 +557,13 @@ cp_size=8
|
||||
3. draft 不生成自己的 owner plan,一切跟随 target。
|
||||
4. target bs>1 ETE 通过后,再打开 draft local path。
|
||||
|
||||
当前实现状态:
|
||||
|
||||
- batch-aware input/position/spec-hidden split 已接到 draft local path;valid CP-local hidden 直接使用,full hidden 走 batch-aware split。
|
||||
- per-request last-token collect 已在 `cp_collect_last_token_hidden()` 中覆盖 batch plan。
|
||||
- 为避免打开 gate 后退回不正确的 full/draft fallback,bs>1 + `SGLANG_CP_DRAFT_SHARED_KV=1` + CP shared-KV 时,draft local path 对 missing/mismatch spec hidden、embedding pad metadata、input embeds shape mismatch 均 fail-fast。
|
||||
- admission gate 仍保持关闭;需要 EAGLE enabled ETE 验证 accept length/output length 后再打开。
|
||||
|
||||
测试重点:
|
||||
|
||||
- `cp_collect_last_token_hidden()` 对 bs=2 返回两个 request 的 last hidden。
|
||||
|
||||
@@ -555,6 +555,18 @@ target path 正确后,再恢复 EAGLE/draft,并做远端 ETE/perf 验证。
|
||||
4. `cp_collect_last_token_hidden()` 返回每个 request 的 last hidden。
|
||||
5. EAGLE accept length 不能因为 cache hit/bs>1 掉到 1。
|
||||
|
||||
### 当前状态
|
||||
|
||||
- target 侧 batch metadata、batch split helper、per-request last-token collect 已有单测覆盖。
|
||||
- EAGLE/draft prefill local path 复用上述 batch-aware split helper:
|
||||
- `cp_split_and_rebuild_1d()` 处理 draft input ids;
|
||||
- `cp_split_and_rebuild_position()` 处理 positions;
|
||||
- `_get_cp_local_spec_hidden_states()` 接受已 CP-local 的 draft hidden,或对 full hidden 走 batch-aware split;
|
||||
- `cp_collect_last_token_hidden()` 按 request owner/offset collect last hidden。
|
||||
- bs>1 时不允许 silent fallback:如果 CP draft shared-KV 已开启但 spec hidden、embedding pad metadata、input embeds 形状不满足 batch fast path,直接
|
||||
`[CP_SHARED_KV_FAIL_FAST][draft_batch_gt1_*]` 报错。bs=1 兼容 fallback 暂时保留。
|
||||
- scheduler 的 bs>1 admission gate 仍未打开;打开前必须完成下面 ETE 场景,尤其是 EAGLE accept length 与 output len。
|
||||
|
||||
### ETE 验证场景
|
||||
|
||||
远端:
|
||||
|
||||
Reference in New Issue
Block a user