Fix CP HiCache L3 reload duplicate-reservation crash: recheck clean-attach at reserve
_drain_l3_control_queues runs validate-candidates -> admit -> reserve in one tick. _cp_l3_admit_reload pops the inflight key and mark_object_committed()s the l3rl object (which stays live in the allocator) while inserting the reloaded node. A second same-suffix candidate, validated against the pre-admit tree, then reaches _cp_l3_reserve_reloads with its dedup guard missing (inflight key gone) and calls allocator.reserve() on the still-live object -> ValueError: duplicate live reservation for object 'l3rl:<hash>' payload 'target_kv' -> scheduler crash. Re-validate _cp_l3_reload_attach_ok in _cp_l3_reserve_reloads after the piggyback check (honoring its documented 'checked at reserve AND at admission' contract, which the original code never enforced). If a same-tick admit inserted the suffix, the candidate is dropped; the un-held request re-matches and hits the just-inserted L2 node. Rank-uniform: attach_ok reads only the replicated radix and admit is rank-uniform, so every CP rank drops the same candidate. The reserve() duplicate raise is kept as a fail-loud backstop for true invariant violations. Confirmed by independent root-cause + adversarial fix review (CLEAN-SHIP): the split-retained-key path that looks residual is provably unreachable via the radix content-path / match_prefix invariant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1553,6 +1553,16 @@ class HiRadixCache(RadixCache):
|
||||
self.ongoing_l3_reload[existing]["waiters"].append(rid) # piggyback on the in-flight reload
|
||||
self.cp_l3_reload_done[rid] = False
|
||||
continue
|
||||
# Re-validate the clean-attach against the POST-admit tree (the contract _cp_l3_reload_attach_ok
|
||||
# documents: "checked at reserve AND at admission -- the tree may change in between"). valid_cands
|
||||
# was built BEFORE _cp_l3_admit_reload ran this tick; a same-tick admit of THIS suffix inserts its
|
||||
# node and pops its inflight key while leaving the l3rl object committed-and-live in the allocator.
|
||||
# Without this recheck the dedup guard above misses (inflight key gone) and reserve() below would
|
||||
# raise "duplicate live reservation" on the still-live object. If the suffix is now present, drop
|
||||
# the candidate; the (un-held) request re-matches and hits the just-inserted L2 node. Rank-uniform
|
||||
# (replicated radix + rank-uniform admit -> every rank skips the same candidate).
|
||||
if not self._cp_l3_reload_attach_ok(lhn, skey):
|
||||
continue
|
||||
if len(self.ongoing_l3_reload) >= self.cp_l3_reload_max_inflight:
|
||||
continue # cap reached (len is replicated) -> recompute
|
||||
ranges = self._cp_l3_reserve_reload_object(reload_key, store.payloads, c, allocator)
|
||||
|
||||
Reference in New Issue
Block a user