chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Stabilize EAGLE draft cache hits under CP HiCache

Browse Source 
The failing runs showed EAGLE accept length collapsing when draft cache-hit suffixes used the new partial-current splice path.  This keeps target partial-current reuse enabled, but returns EAGLE/NextN draft cache-hit suffixes to the previous full-materialize path with an explicit fallback warning until the draft splice path has value-level ETE proof.\n\nThe same change set also tightens the page-granular CP HiCache contract for scheduler-visible hits and makes the prefill-to-decode EAGLE handoff observable without cloning hot-path metadata.  Exact non-page CP hits are floored to a page boundary for new scheduling decisions, while internal unfinished-request refresh keeps its exact accounting.\n\nConstraint: CP shared KV and HiCache operate at page granularity; exposing token-precise CP tails to scheduler-visible cache hits can force non-page partial materialization.\nConstraint: EAGLE/NextN draft has only one executable layer, so draft prefetch and draft partial-current splice need a separate correctness contract from target layers.\nRejected: Keep draft partial-current splice enabled | remote logs correlate it with avg accept length around 0.068 and median 0.\nRejected: Clone decode metadata tensors on transfer | slot ownership until process_prebuilt consumes them avoids extra hot-path copies.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not re-enable draft partial-current reuse without metadata/draft-KV value checks and ETE accept-length evidence.\nTested: g0034 container py_compile for touched modules.\nTested: g0034 container PYTHONPATH=python python -m pytest -q test/registered/unit/disaggregation/test_decode_queue_compaction.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 183 passed, 5 warnings, 2 subtests passed.\nNot-tested: Fresh ETE accept-length run after this exact commit; requires user-driven traffic restart.
This commit is contained in:
laoyao0822
2026-05-30 22:31:43 +08:00
parent 10296a5fef
commit b328baec7c
14 changed files with 920 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
test/registered/unit/disaggregation/test_decode_queue_compaction.py
View File

@@ -428,6 +428,9 @@ class TestDecodeQueueCompaction(CustomTestCase):
allocator = FakeAllocator()
req = FakeReq("eagle", 3)
req.metadata_buffer_index = 9
req.output_topk_p = torch.ones((1,), dtype=torch.float32)
req.output_topk_index = torch.ones((1,), dtype=torch.int64)
req.hidden_states_tensor = torch.ones((4,), dtype=torch.float32)
scheduler = SchedulerOutputProcessorMixin.__new__(SchedulerOutputProcessorMixin)
scheduler.req_to_metadata_buffer_idx_allocator = allocator
@@ -437,6 +440,9 @@ class TestDecodeQueueCompaction(CustomTestCase):
self.assertEqual(allocator.freed, [9])
self.assertEqual(req.metadata_buffer_index, -1)
self.assertIsNone(req.output_topk_p)
self.assertIsNone(req.output_topk_index)
self.assertIsNone(req.hidden_states_tensor)
def test_resume_retracted_reqs_compacts_queue_in_one_pass(self):
prealloc_queue = DecodePreallocQueue.__new__(DecodePreallocQueue)
@@ -689,6 +695,9 @@ class TestDecodeQueueCompaction(CustomTestCase):
self.assertEqual(allocator.freed, [20, 21, 22])
for req in captured["reqs"]:
self.assertEqual(req.metadata_buffer_index, -1)
self.assertIsNone(getattr(req, "output_topk_p", None))
self.assertIsNone(getattr(req, "output_topk_index", None))
self.assertIsNone(getattr(req, "hidden_states_tensor", None))
def test_get_new_prebuilt_batch_keeps_waiting_queue_when_no_capacity(self):
scheduler = cast(Any, SimpleNamespace())