Gate CP HiCache write-backup on hard deficit, not the free-room watermark

A long cold-request workload flooded the prefill log (~177MB / 32K lines in
8min, all 8 CP ranks) with cp_host_reservation_plan_insufficient +
prepare_write_backup_reservation_failed even though the host pool was <42%
used and the writes physically fit.

Root cause: _free_room_deficit folds the proactive 20% free-room target
(hicache_host_free_room_ratio) into the admission deficit once a lane dips
below the 10% trigger. _evict_cp_host_for_write_admission then failed -- and
evicted nothing -- whenever it could not reclaim that full 20% target, which
is impossible while the residual is locked/pending. The lane never drained,
so every subsequent backup re-issued the same impossible demand and re-logged
on every tick. The reservation failure is itself a graceful skip, so there is
no crash; the symptom is the log storm.

Gate write-backup admission on the HARD deficit -- max(0, required-available)
over the target and draft lanes (draft_available is 2**62 with no draft pool,
a no-op) -- and evict toward the watermark best-effort, admitting whenever the
write itself fits. This drains the stuck lane and backs the request up instead
of skipping+flooding. The proactive watermark stays as the eviction target
(deficit_by_owner unchanged); it is no longer a hard admission gate. The
decision is computed from rank-replicated state on the collective-free reserve
path, so every CP rank makes the same admit/skip choice (opus-reviewed
rank-safe).

Also rate-limit the four host-reservation fallback warnings (once per 10s per
fallback_name, with a suppressed count) so any genuine exhaustion degrades
quietly instead of producing a log storm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test/registered/unit/mem_cache/test_cp_hicache_metadata.py

@@ -805,6 +805,79 @@ class TestCpHiCacheFreeRoom(CustomTestCase):
         # required=128, target_room=128, available=96.
         self.assertEqual(admission.deficit_by_owner, (128, 0))
 
+    def test_evict_admission_admits_write_that_fits_below_watermark(self):
+        # Regression (2026-06-19 host-reservation log storm): a backup that
+        # physically fits (required <= available, hard deficit 0) must be ADMITTED
+        # even when the lane is below the proactive free-room watermark and that
+        # advisory eviction target (deficit_by_owner / remaining_deficit) cannot be
+        # met. The old gate keyed off remaining_deficit and returned False here,
+        # skipping the backup AND re-firing the warning every tick on every rank.
+        cache = HiRadixCache.__new__(HiRadixCache)
+        admission = CpWriteAdmission(
+            node_id=1,
+            phase="unit",
+            required_by_owner=(100, 100),
+            target_available_by_owner=(500, 500),  # write fits -> hard deficit 0
+            draft_available_by_owner=(2**62, 2**62),  # no draft pool sentinel
+            deficit_by_owner=(700000, 700000),  # proactive watermark target
+            eviction_plan=CpHiCacheEvictionPlan(
+                victims=(),
+                planned_freed=(0, 0),
+                remaining_deficit=(700000, 700000),  # watermark unreachable
+            ),
+        )
+        self.assertTrue(
+            cache._evict_cp_host_for_write_admission(
+                admission, node_id=1, phase="unit"
+            )
+        )
+
+    def test_evict_admission_fails_when_write_does_not_fit(self):
+        # When the backup genuinely does not fit (required > available) and
+        # eviction cannot free enough, admission must still fail (skip the backup).
+        cache = HiRadixCache.__new__(HiRadixCache)
+        admission = CpWriteAdmission(
+            node_id=2,
+            phase="unit",
+            required_by_owner=(1000, 0),
+            target_available_by_owner=(100, 500),  # hard deficit 900 on lane 0
+            draft_available_by_owner=(2**62, 2**62),
+            deficit_by_owner=(900, 0),
+            eviction_plan=CpHiCacheEvictionPlan(
+                victims=(),
+                planned_freed=(0, 0),  # freed nothing
+                remaining_deficit=(900, 0),
+            ),
+        )
+        self.assertFalse(
+            cache._evict_cp_host_for_write_admission(
+                admission, node_id=2, phase="unit"
+            )
+        )
+
+    def test_evict_admission_admits_when_eviction_covers_hard_deficit_only(self):
+        # Eviction that reaches the HARD deficit (required-available) but not the
+        # full proactive watermark target must still admit (best-effort drain).
+        cache = HiRadixCache.__new__(HiRadixCache)
+        admission = CpWriteAdmission(
+            node_id=3,
+            phase="unit",
+            required_by_owner=(1000, 0),
+            target_available_by_owner=(100, 500),  # hard deficit 900 on lane 0
+            draft_available_by_owner=(2**62, 2**62),
+            deficit_by_owner=(700000, 0),  # watermark target
+            eviction_plan=CpHiCacheEvictionPlan(
+                victims=(),
+                planned_freed=(900, 0),  # freed exactly the hard deficit
+                remaining_deficit=(699100, 0),  # watermark not reached
+            ),
+        )
+        self.assertTrue(
+            cache._evict_cp_host_for_write_admission(
+                admission, node_id=3, phase="unit"
+            )
+        )
+
     def test_prepare_write_backups_for_reqs_runs_one_batch_admission(self):
         cache = HiRadixCache.__new__(HiRadixCache)
         cache.disable = False