Count evictable device cache when gating HiCache load-back
HiCache host hits can be skipped before load-back when the quota gate only counts immediately free KV allocator space. Under CP shared-KV pressure most reusable capacity may be represented as evictable radix-cache leaves, so the gate can incorrectly reject a host hit and leave prefill with cached-token zero despite host residency. Count device evictable cache in the quota estimate while leaving actual owner-lane allocation and eviction checks in the load path. Constraint: CP HiCache load-back still has to respect owner-lane allocation and allocator eviction semantics. Rejected: Force load-back regardless of quota | would bypass the scheduler pressure signal and increase OOM risk. Rejected: Treat cache-hit zero as a transfer issue | logs showed host hits were found but skipped by quota before transfer. Confidence: medium Scope-risk: moderate Directive: Do not remove evictable cache from load-back capacity accounting without checking CP HiCache host-hit behavior under device pressure. Tested: git diff --check Tested: remote g0034 container pytest -q test/registered/unit/managers/test_prefill_adder.py test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_alloc_pages_with_owners.py (90 passed, 3 warnings) Not-tested: Full ETE GLM5 CP+HiCache+EAGLE pressure run after this quota change Co-authored-by: OmX <omx@oh-my-codex.dev>
This commit is contained in:
@@ -501,8 +501,19 @@ class PrefillAdder:
|
||||
|
||||
def _get_available_device_tokens_for_load_back(self) -> int:
|
||||
if self.is_hybrid_swa:
|
||||
return self.token_to_kv_pool_allocator.full_available_size()
|
||||
return self.token_to_kv_pool_allocator.available_size()
|
||||
return (
|
||||
self.token_to_kv_pool_allocator.full_available_size()
|
||||
+ self.tree_cache.full_evictable_size()
|
||||
)
|
||||
if self.is_hybrid_ssm_cache:
|
||||
return (
|
||||
self.token_to_kv_pool_allocator.available_size()
|
||||
+ self.tree_cache.full_evictable_size()
|
||||
)
|
||||
return (
|
||||
self.token_to_kv_pool_allocator.available_size()
|
||||
+ self.tree_cache.evictable_size()
|
||||
)
|
||||
|
||||
def _get_load_back_mem_quota(self, real_input_tokens: int) -> int:
|
||||
reserve_tokens = real_input_tokens + self.page_size
|
||||
|
||||
Reference in New Issue
Block a user