From 8cd46c18b320609b27f3c0121d2fc35cd0a6098f Mon Sep 17 00:00:00 2001 From: leavelet Date: Wed, 24 Jun 2026 14:48:37 +0000 Subject: [PATCH] Fix CP HiCache L3 reload duplicate-reservation crash: recheck clean-attach at reserve _drain_l3_control_queues runs validate-candidates -> admit -> reserve in one tick. _cp_l3_admit_reload pops the inflight key and mark_object_committed()s the l3rl object (which stays live in the allocator) while inserting the reloaded node. A second same-suffix candidate, validated against the pre-admit tree, then reaches _cp_l3_reserve_reloads with its dedup guard missing (inflight key gone) and calls allocator.reserve() on the still-live object -> ValueError: duplicate live reservation for object 'l3rl:' payload 'target_kv' -> scheduler crash. Re-validate _cp_l3_reload_attach_ok in _cp_l3_reserve_reloads after the piggyback check (honoring its documented 'checked at reserve AND at admission' contract, which the original code never enforced). If a same-tick admit inserted the suffix, the candidate is dropped; the un-held request re-matches and hits the just-inserted L2 node. Rank-uniform: attach_ok reads only the replicated radix and admit is rank-uniform, so every CP rank drops the same candidate. The reserve() duplicate raise is kept as a fail-loud backstop for true invariant violations. Confirmed by independent root-cause + adversarial fix review (CLEAN-SHIP): the split-retained-key path that looks residual is provably unreachable via the radix content-path / match_prefix invariant. Co-Authored-By: Claude Opus 4.8 (1M context) --- python/sglang/srt/mem_cache/hiradix_cache.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/python/sglang/srt/mem_cache/hiradix_cache.py b/python/sglang/srt/mem_cache/hiradix_cache.py index 9a7f620fc..ad3d2b57d 100644 --- a/python/sglang/srt/mem_cache/hiradix_cache.py +++ b/python/sglang/srt/mem_cache/hiradix_cache.py @@ -1553,6 +1553,16 @@ class HiRadixCache(RadixCache): self.ongoing_l3_reload[existing]["waiters"].append(rid) # piggyback on the in-flight reload self.cp_l3_reload_done[rid] = False continue + # Re-validate the clean-attach against the POST-admit tree (the contract _cp_l3_reload_attach_ok + # documents: "checked at reserve AND at admission -- the tree may change in between"). valid_cands + # was built BEFORE _cp_l3_admit_reload ran this tick; a same-tick admit of THIS suffix inserts its + # node and pops its inflight key while leaving the l3rl object committed-and-live in the allocator. + # Without this recheck the dedup guard above misses (inflight key gone) and reserve() below would + # raise "duplicate live reservation" on the still-live object. If the suffix is now present, drop + # the candidate; the (un-held) request re-matches and hits the just-inserted L2 node. Rank-uniform + # (replicated radix + rank-uniform admit -> every rank skips the same candidate). + if not self._cp_l3_reload_attach_ok(lhn, skey): + continue if len(self.ongoing_l3_reload) >= self.cp_l3_reload_max_inflight: continue # cap reached (len is replicated) -> recompute ranges = self._cp_l3_reserve_reload_object(reload_key, store.payloads, c, allocator)