chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Preserve transferred EAGLE state past metadata-slot reuse

Browse Source 
Decode committed EAGLE top-k and hidden-state tensors as views into reusable metadata-buffer rows. The metadata index is freed immediately after transfer commit, while the request may wait before process_prebuilt consumes the draft state. Under concurrent cache-hit traffic a later transfer can overwrite the same row, leaving output_id copied correctly but EAGLE draft state corrupted, which matches low accept length despite successful KV/state registration.

Constraint: Metadata slots are intentionally recycled right after transfer commit for throughput.

Rejected: Hold metadata slots until process_prebuilt | larger lifetime change and reduces transfer capacity; cloning the small prebuilt EAGLE state is narrower.

Confidence: high

Scope-risk: narrow

Directive: Do not store reusable metadata-buffer views on Req unless the slot lifetime is extended through all consumers.

Tested: Local py_compile for decode.py and test_decode_queue_compaction.py.

Tested: Remote g0034 container py_compile for decode.py and test_decode_queue_compaction.py.

Tested: Remote g0034 focused clone-lifetime test: 1 passed.

Tested: Remote g0034 test_decode_queue_compaction.py: 10 passed, 5 warnings.

Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
This commit is contained in:
laoyao0822
2026-05-30 03:36:42 +08:00
parent b7364d23f9
commit 07c9544737
3 changed files with 121 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md
+35
View File
@@ -2188,3 +2188,38 @@ Checked point:
Rejected:
- Do not change prebuilt `num_states` to `topk * speculative_num_steps` for the non-multi-layer EAGLE path unless the launch also enables `enable_multi_layer_eagle`. That would feed extra uninitialized/irrelevant metadata into `EAGLEWorkerV2.draft()`.
### C45. 2026-05-30 new runtime did not recover; root-cause moved to decode metadata-buffer lifetime
Runtime evidence:
- New prefill process PID `2655723` started at `2026-05-30 03:18:38 CST`, after the C42 code sync.
- Latest logs:
- prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_191838.log`
- decode0: `/mnt/beegfs/cjy/log/decode0_20260529_191842.log`
- decode1: `/mnt/beegfs/cjy/log/decode1_20260529_191845.log`
- The C42 change is active enough to remove the old fallback: `draft_partial_current_reuse_disabled=0` in the new prefill log.
- Accept length still did not recover: parsed decode batches show average accept length about `1.35`, median about `1.25`, and many windows at `<=1.05`. Transfer-free windows remain low as well.
New root cause:
- `DecodeTransferQueue._commit_transfer_to_req()` assigned `req.output_topk_p`, `req.output_topk_index`, and `req.hidden_states_tensor` directly from `metadata_buffers.get_buf(idx)`.
- `metadata_buffers.get_buf(idx)` returns views into reusable metadata-buffer rows.
- `DecodeTransferQueue.pop_transferred()` frees `metadata_buffer_index` immediately after `_commit_transfer_to_req()` returns, but the transferred request can sit in the scheduler waiting queue before `process_prebuilt()` consumes the EAGLE top-k/hidden state.
- Therefore later transfers can overwrite the same metadata slot before decode prebuilt consumes it. The scalar output id and cached-token count are copied with `.item()`, but the EAGLE draft state was not owned. This explains why transfer and registration look correct while the first decode draft state is often wrong under concurrent/high-cache-hit traffic.
Fix:
- Clone the EAGLE metadata tensors at commit time before freeing the metadata slot:
- `output_topk_p.clone()`
- `output_topk_index.clone()`
- `output_hidden_states.clone()`
- Added a unit test that mutates the original metadata-buffer tensors after commit and verifies the request still owns the original values and different storage.
- Remote container verification on `g0034:/sgl-workspace/sglang-tai` passed:
- py_compile for `decode.py` and `test_decode_queue_compaction.py`
- focused clone-lifetime test: `1 passed`
- full `test_decode_queue_compaction.py`: `10 passed, 5 warnings`
Guardrail:
- Do not replace the clone with a view unless the allocator lifetime is changed so the metadata slot is held until after `process_prebuilt()` consumes the request.