CP HiCache: replicated-clock SLRU eviction, collective-safe (Phase 1)

Make CP shared-KV HiCache eviction collective-free and scan-resistant by
removing every per-rank wall-clock input from the eviction/admission decisions
(any such input desyncs the must-be-replicated victim set / batch across the 8
CP ranks and can deadlock the collective-coupled writeback/load-back).

- Replicated logical clock: TreeNode.next_access_time() (a process-global
  monotonic counter, mirrors mamba/swa's get_last_access_time) replaces
  time.monotonic() as the source of last_access_time at every radix bump site
  (radix_cache __init__/match/insert; hiradix CP match/insert/_insert_helper_host;
  reset() zeroes it). creation_time/pin_expiry intentionally stay wall-clock.
  Because match/insert events are replicated (reqs broadcast from rank 0 over the
  replicated tree), last_access_time is now identical on every rank. Unique ints
  also give a strict total order (no LRU tie ambiguity). No duration arithmetic
  reads last_access_time, so non-CP LRU is unchanged; mamba/swa use their own
  TreeNode and are untouched.
- CpReplicatedSLRUStrategy = (is_protected[hit>=2], last_access_time, node.id):
  scan-resistant (cold one-shots stay probationary, evicted before reused/
  protected prefixes), recency-within-segment ages out cold (not LFU), node.id
  is the deterministic total-order tiebreak (heaps never compare TreeNodes).
  Overridden for CP so it reaches every get_priority eviction site; the host
  write-admission key _cp_host_evict_key is switched from FIFO-by-id to it.
- Co-fixes for the same per-rank-wall-clock class: pin_prefix is forbidden under
  CP (its pin_expiry victim-eligibility check is per-rank wall-clock; SLRU
  scan-resistance covers the benefit); the affinity head_age_s admission input is
  disabled (=0.0, relying on the replicated head_defer_count bound); and a
  server_args fail-fast guards CP shared-KV against SGLANG_REQ_WAITING_TIMEOUT>0
  (its waiting-queue pruning is per-rank wall-clock).

Tests: test_evict_policy.py adds TestCpReplicatedSLRUStrategy (scan-resistance,
recency, total-order, full ordering) -- 30 passed in the dev-cu13 container;
test_radix_cache_unit.py adds logical-clock determinism/total-order tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test/registered/unit/mem_cache/test_radix_cache_unit.py

@@ -17,12 +17,15 @@ Usage:
     python -m pytest test_radix_cache_unit.py::TestRadixCache::test_insert_basic
 """
 
-import sys
 import unittest.mock
 
-for _mod in ("sgl_kernel", "sgl_kernel.kvcacheio"):
-    if _mod not in sys.modules:
-        sys.modules[_mod] = unittest.mock.MagicMock()
+# NB: do NOT stub sgl_kernel here. This test is registered GPU-only
+# (register_cuda_ci/register_amd_ci), so the real sgl_kernel is always present,
+# and the import chain touches sgl_kernel solely through fp8_kernel.py under
+# `if _is_cuda:`. A MagicMock would satisfy `from sgl_kernel import ...` while
+# leaving the real C++ ops unregistered, so fp8_kernel's module-level
+# torch.library.register_fake("sgl_kernel::...") would raise
+# "operator sgl_kernel::... does not exist".
 
 from sglang.srt.mem_cache.common import available_and_evictable_str
 from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
@@ -235,12 +238,33 @@ class TestTreeNode(unittest.TestCase):
     def test_lt_comparison(self):
         """Test less than comparison based on last_access_time."""
         node1 = TreeNode()
-        time.sleep(0.001)  # Small delay to ensure different timestamps
+        time.sleep(0.001)  # (now a no-op: last_access_time is the logical clock)
         node2 = TreeNode()
 
         self.assertTrue(node1 < node2)
         self.assertFalse(node2 < node1)
 
+    def test_next_access_time_is_monotonic_logical_clock(self):
+        """next_access_time() is a replicated monotonic logical counter (NOT
+        wall-clock): consecutive calls strictly increase by 1 and return ints.
+        Under CP this is what makes last_access_time rank-replicated."""
+        a = TreeNode.next_access_time()
+        b = TreeNode.next_access_time()
+        c = TreeNode.next_access_time()
+        self.assertIsInstance(a, int)
+        self.assertEqual(b, a + 1)
+        self.assertEqual(c, b + 1)
+
+    def test_creation_order_is_strict_total_order(self):
+        """Nodes take last_access_time from the logical clock at construction, so
+        creation order is a strict total order with no wall-clock dependence or
+        ties (no sleep needed)."""
+        n1 = TreeNode()
+        n2 = TreeNode()
+        n3 = TreeNode()
+        self.assertLess(n1.last_access_time, n2.last_access_time)
+        self.assertLess(n2.last_access_time, n3.last_access_time)
+
 
 class TestTreeNodeChildren(CustomTestCase):
     def test_missing_child_access_does_not_create_orphan_node(self):