# NSA Prefill CP HiCache Reactive Host Free-Room Plan

Date: 2026-06-02

## Goal

Keep CP HiCache host/L2 from running at the exact-full boundary.  When a host
write reservation discovers that capacity is insufficient, evict enough host
cache so the reservation can complete and a configured free-room watermark
remains after the reservation.

This is intentionally **reactive**.  It must not resurrect the old proactive
per-scheduler-tick host eviction loop.

## Current eviction behavior

### L1/device cache eviction

Entry points:

- `python/sglang/srt/mem_cache/common.py::evict_from_tree_cache`
- `python/sglang/srt/mem_cache/common.py::_evict_for_compute_owner_lanes`
- `python/sglang/srt/mem_cache/hiradix_cache.py::HiRadixCache.evict`

Current behavior:

- Standard device allocator path checks `available < num_tokens`.
- If insufficient, it calls `tree_cache.evict(EvictParams(num_tokens=num_tokens))`.
- CP owner-lane path computes physical owner-lane page deficits and evicts
  `deficit_pages * page_size`, without the old `* cp_size` over-eviction.
- `HiRadixCache.evict` picks victims from `evictable_leaves` by priority:
  - host-backed nodes are demoted by freeing device slots only;
  - unbacked nodes are either backed up first under `write_back` or deleted;
  - eviction stops after `num_evicted >= params.num_tokens`.

There is no stable L1 free-room watermark in the current branch.

### L2/host cache eviction

Entry points:

- `python/sglang/srt/mem_cache/hiradix_cache.py::HiRadixCache.evict_host`
- `python/sglang/srt/mem_cache/hiradix_cache.py::_evict_host_for_physical_slots`

Current behavior:

- CP HiCache host eviction frees host-only, backed, unpinned, unreferenced host
  leaves.
- CP target and draft host slots are freed together through
  `cache_controller.evict_cp_host(...)`.
- Eviction stops after the exact requested physical slot count is freed.

There is no host/L2 free-room watermark in the current branch.

### CP write-admission host reservation

Entry points:

- `python/sglang/srt/mem_cache/hiradix_cache.py::_cp_build_write_admission`
- `python/sglang/srt/mem_cache/hiradix_cache.py::_plan_cp_host_evictions`
- `python/sglang/srt/mem_cache/hiradix_cache.py::_reserve_write_cp_indices_no_collective`

Current behavior:

```text
required_by_owner = required host tokens/pages for this write
target_deficit    = max(0, required_by_owner - target_available_by_owner)
draft_deficit     = max(0, required_by_owner - draft_available_by_owner)
deficit           = max(target_deficit, draft_deficit)
```

The eviction plan only covers the exact per-owner deficit.  After a successful
reservation, the affected owner lanes can be left near zero free host slots,
which makes later writes repeatedly enter the eviction path.

## Historical implementation to avoid

The old free-room work around commit `579fe3d41` maintained host room with a
proactive "Round 2.5" scheduler-tick eviction path.  That direction is not the
desired fix here because it can:

- scan and evict on every scheduler tick;
- introduce hot-path synchronization pressure;
- evict repeatedly under sustained host pressure even if the next reservation
  would still fit;
- recreate the earlier performance-regression shape.

The new design should not add a per-tick maintenance loop.

## Proposed design

Add a **reactive host free-room watermark** to CP HiCache write admission.

Initial draft, superseded below, only evicted when a reservation would not fit:

```text
if required <= available:
    deficit = 0
```

When a reservation would not fit, evict enough for the write and for post-write
free room:

```text
if required > available:
    deficit = required + free_room_target - available
```

This `required + free_room_target` target is now rejected because it can
over-reserve free room and waste cache residency under many small cache-hit
requests.

## Scope for the first implementation

In scope:

1. CP HiCache host/L2 write-admission path.
2. Per CP owner-lane free-room target.
3. Target and draft host pools must both satisfy the same free-room contract.
4. Explicit configuration for the free-room ratio.
5. Unit tests for admission planning and target/draft symmetry.

Out of scope:

1. L1/device free-room changes.
2. Load-back owner-lane device eviction changes.
3. Per-tick proactive host eviction.
4. Allocator contiguity changes.
5. Evict victim merge/sort policy changes.
6. New collectives/all-reduce.

## Configuration

Preferred first version:

- Add `--hicache-host-free-room-ratio`.
- Default: `0.0` to preserve current behavior.
- Remote experiments can explicitly set a small value such as `0.03` or `0.05`.

The target should be rounded to page units so CP host accounting remains
page-granular.

## Implementation notes

Likely code points:

1. Add `hicache_host_free_room_ratio` to `ServerArgs`.
2. Store the ratio in `HiRadixCache`.
3. Add helper to compute per-owner host free-room targets from
   `_cp_host_capacity_snapshot()`.
4. Change `_cp_build_write_admission()` so host deficits become reactive
   low-watermark deficits instead of exact deficits.
5. Keep `_plan_cp_host_evictions()` unchanged initially: it already accepts a
   per-owner deficit vector and picks deterministic host victims.
6. Keep `_reserve_write_cp_indices_no_collective()` unchanged except for using
   the updated admission result.

Important detail:

The free-room target should only be applied when that pool/lane is already
short for the current write.  Do not evict merely because `available` is below
`free_room_target` if the current reservation still fits.

## Expected effect

Compared with exact-deficit eviction:

- fewer repeated host evictions when host memory is near full;
- less scheduler/control-path churn under sustained write pressure;
- better chance for later reservations to complete without immediate eviction;
- no extra collective cost.

Compared with old proactive free-room maintenance:

- lower steady-state overhead;
- eviction work only happens on real reservation pressure;
- no recurring scheduler-tick eviction loop.

## Test plan

Unit tests:

1. `ratio=0.0` preserves exact-deficit behavior.
2. If `required <= available`, no free-room eviction is planned.
3. If `required > available`, planned deficit includes `required + room - available`.
4. Target/draft symmetry: deficit uses the max of target and draft low-watermark
   requirements.
5. Rounding: free-room target is page-aligned.

Runtime validation:

1. Start with default ratio `0.0` to confirm no behavior change.
2. Run remote ETE with `--hicache-host-free-room-ratio 0.03`.
3. Check host eviction frequency and host free-slot floor in logs.
4. Verify no new CP HiCache collective logs are introduced.

## Risks

1. Too large a ratio can evict useful host cache and lower hit rate.
2. Evicting large nodes may overshoot the target; this is acceptable initially
   because host eviction is node/page based.
3. If host pressure is caused by pinned or in-flight backup nodes, free-room
   eviction may still fail; failures must stay explicit and warning-level.
4. The first version does not improve physical page contiguity; it only reduces
   eviction frequency.

## 2026-06-02 correction: trigger and target are separate

The first draft only added target room when `required > available`.  The current
accepted contract separates **when** eviction starts from **how far** eviction
continues:

```text
if available >= required + trigger_room:
    deficit = 0
else:
    deficit = max(max(0, required - available), target_room)
```

The trigger is evaluated after accounting for the pending allocation
(`available - required < trigger_room`).  Once triggered, `target_room` is the
minimum eviction chunk; if the current allocation needs more than that, evict the
larger exact deficit.  It deliberately does **not** restore `required +
target_room`, because doing so would leave a full target room after the current
allocation and can evict useful cache too aggressively.

Implications:

- `trigger_room=0` preserves the conservative behavior: evict only when the
  current reservation would not fit.
- `target_room>0` makes a triggered eviction release extra pages.
- L1/device and L2/host should use the same trigger/target model, but with
  separate ratios because L1 hit value and host capacity pressure are different.
- Free-room alone is not sufficient for contiguous allocation. `HostKVCache` and
  CP owner-lane device allocation also need contiguous-preferred selection.

## 2026-06-05 correction: load-back must not synchronously evict for L1 free room

Remote evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260605_152940.log`
showed prefill slowdown and death caused by a CP0 HiCache load-back planning
straggler:

```text
node_id=677 load_back CP starts on all CP ranks at 15:48:26
CP1-7 report SUCCESS by 15:48:27
CP0 reports SUCCESS at 15:48:49
health check fires at 15:48:47 due no detokenizer heartbeat for 20s
```

Important detail: `HiCache-load load_back CP SUCCESS` is emitted before
`cache_controller.start_loading()`, so this 23s gap is not the actual L2->L1 H2D
copy.  It is in the scheduler-side admission/planning path:

```text
HiRadixCache.load_back
  -> _build_cp_load_back_plan
  -> optional _evict_cp_load_back_owner_lanes
  -> HiCacheController.load_cp
      -> alloc_pages_with_owners
      -> build host/device index descriptors
```

The failing node had exact capacity available but non-zero L1 free-room deficit:

```text
required_by_owner=[41,45,45,45,46,46,46,46]
available_by_owner=[221,194,197,198,237,285,290,308]
free-room deficit on lanes 1-3
```

Therefore the reactive free-room rule must be different for synchronous
load-back admission:

- exact owner-lane deficit is blocking and may trigger synchronous eviction;
- free-room deficit is advisory/observability for load-back and must not force
  synchronous eviction when exact capacity already fits.

This preserves correctness while avoiding heavy eviction planning on the cache-hit
scheduler hot path.  Free-room maintenance for L1 should happen on real extend
allocation pressure or a later non-blocking/proactive path, not inside a
cache-hit load-back that already has enough exact capacity.
