Reuse draft MTP indexer topk on spec v1 EAGLE
Spec v1 EAGLE now follows the same model-config-gated MTP index reuse contract as spec v2. This lets models that declare index_share_for_mtp_iteration reuse the first draft step's NSA/DSA topk indices across internal MTP iterations while keeping the unsafe topk>1 case disabled. Constraint: select_top_k_tokens can reorder hidden rows when topk > 1, so carried topk indices are only valid under topk == 1. Rejected: Enable reuse unconditionally | models without the config flag may not have compatible MTP index semantics. Rejected: Broaden to target-to-draft index reuse | separate semantic change with different correctness risks. Confidence: high Scope-risk: narrow Directive: Keep spec v1 and spec v2 MTP index reuse semantics aligned, including the topk==1 guard and per-draft-forward cleanup. Tested: python -m pytest test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py -q Tested: python -m py_compile python/sglang/srt/speculative/eagle_worker.py test/registered/spec/eagle/test_eagle_v2_draft_extend_contract.py Not-tested: full spec v1 GLM5 online throughput run
This commit is contained in:
@@ -156,6 +156,19 @@ class EAGLEWorker(TpModelWorker):
|
||||
memory_pool_config=target_worker.model_runner.memory_pool_config,
|
||||
)
|
||||
|
||||
# Reuse the first draft step's NSA/DSA indexer topk across the rest of
|
||||
# the MTP iteration when the model config says it is safe. The reuse is
|
||||
# only valid for topk == 1: select_top_k_tokens reorders rows for topk
|
||||
# > 1, which would desynchronize carried indices from hidden states.
|
||||
self.index_share_for_mtp_iteration = (
|
||||
getattr(
|
||||
self.draft_model_runner.model_config.hf_config,
|
||||
"index_share_for_mtp_iteration",
|
||||
False,
|
||||
)
|
||||
and self.topk == 1
|
||||
)
|
||||
|
||||
embed, head = self.target_worker.model_runner.model.get_embed_and_head()
|
||||
|
||||
if self.speculative_algorithm.is_eagle3():
|
||||
@@ -701,6 +714,10 @@ class EAGLEWorker(TpModelWorker):
|
||||
token_list: List[torch.Tensor] = []
|
||||
parents_list: List[torch.Tensor] = []
|
||||
|
||||
if self.index_share_for_mtp_iteration:
|
||||
forward_batch.reuse_mtp_topk_indices = True
|
||||
forward_batch.topk_indices = None
|
||||
|
||||
# Forward multiple steps
|
||||
scores = None
|
||||
for i in range(self.speculative_num_steps):
|
||||
@@ -751,6 +768,10 @@ class EAGLEWorker(TpModelWorker):
|
||||
score_list, token_list, parents_list, self.speculative_num_draft_tokens
|
||||
)
|
||||
|
||||
if self.index_share_for_mtp_iteration:
|
||||
forward_batch.topk_indices = None
|
||||
forward_batch.reuse_mtp_topk_indices = False
|
||||
|
||||
return parent_list, top_scores_index, draft_tokens
|
||||
|
||||
def clear_cache_pool(self):
|
||||
|
||||
Reference in New Issue
Block a user