chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Preserve transferred EAGLE state past metadata-slot reuse

Browse Source 
Decode committed EAGLE top-k and hidden-state tensors as views into reusable metadata-buffer rows. The metadata index is freed immediately after transfer commit, while the request may wait before process_prebuilt consumes the draft state. Under concurrent cache-hit traffic a later transfer can overwrite the same row, leaving output_id copied correctly but EAGLE draft state corrupted, which matches low accept length despite successful KV/state registration.

Constraint: Metadata slots are intentionally recycled right after transfer commit for throughput.

Rejected: Hold metadata slots until process_prebuilt | larger lifetime change and reduces transfer capacity; cloning the small prebuilt EAGLE state is narrower.

Confidence: high

Scope-risk: narrow

Directive: Do not store reusable metadata-buffer views on Req unless the slot lifetime is extended through all consumers.

Tested: Local py_compile for decode.py and test_decode_queue_compaction.py.

Tested: Remote g0034 container py_compile for decode.py and test_decode_queue_compaction.py.

Tested: Remote g0034 focused clone-lifetime test: 1 passed.

Tested: Remote g0034 test_decode_queue_compaction.py: 10 passed, 5 warnings.

Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
This commit is contained in:
laoyao0822
2026-05-30 03:36:42 +08:00
parent b7364d23f9
commit 07c9544737
3 changed files with 121 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
python/sglang/srt/disaggregation/decode.py
View File

@@ -1175,9 +1175,15 @@ class DecodeTransferQueue:
decode_req.req.output_ids.append(output_id[0].item())
decode_req.req.cached_tokens = cached_tokens[0].item()
if not self.spec_algorithm.is_none():
decode_req.req.output_topk_p = output_topk_p
decode_req.req.output_topk_index = output_topk_index
decode_req.req.hidden_states_tensor = output_hidden_states
# ``metadata_buffers.get_buf`` returns views into the reusable
# metadata slots. ``pop_transferred`` frees the slot immediately
# after this commit, while the request can stay in the waiting queue
# before ``process_prebuilt`` consumes the EAGLE state. Keep an
# owned copy so subsequent transfers cannot overwrite the prebuilt
# draft top-k/hidden state.
decode_req.req.output_topk_p = output_topk_p.clone()
decode_req.req.output_topk_index = output_topk_index.clone()
decode_req.req.hidden_states_tensor = output_hidden_states.clone()
_cp_draft_shared_kv_debug(
"decode_transfer_commit rid=%s room=%s metadata_idx=%s cached_tokens=%s "