Batch CP HiCache host admission before write reservation

CP shared-KV prefill batching can present multiple requests to HiCache write-through in one forward. Keeping host free-room admission per request causes repeated small L2 evictions and defeats the free-room policy, while request metadata and radix attachment still need per-request ownership.\n\nThis change adds a batch prepare path that aggregates required host tokens by CP owner lane, performs one host admission/eviction step for the batch, and then reserves/submits each request independently. SessionAwareCache and the scheduler prefer the batch API when available.\n\nConstraint: Radix nodes, host slots, draft slots, rollback, and ack semantics remain per request.\nRejected: Merge reservations or radix nodes across requests | would complicate rollback and writing_check ack ownership.\nRejected: Add collective synchronization for host admission | local owner-lane logic already provides deterministic capacity accounting.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce per-request host free-room eviction in the scheduler path without profiling host-full workloads.\nTested: local py_compile for touched Python files\nTested: remote g0034 docker PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 117 passed, 5 warnings\nNot-tested: full ETE bs>1 HiCache run under production traffic
This commit is contained in:
laoyao0822
2026-06-04 04:32:40 +08:00
parent 108e7d866d
commit 02af370e87
5 changed files with 342 additions and 27 deletions

View File

@@ -935,3 +935,61 @@ scheduler admission 不做 bf16/fp8 字节级估算。原因是实际 KV pool
1. `cp_shared_kv_prefill_max_total_extend_tokens` 首版使用 page-aligned extend 累计,和 allocator 的最小 page 单位一致;日志/指标里如果要展示用户侧 token需要另加 valid-token 统计。
2. 该 gate 只控制 scheduler 组 batch不解决所有 bs>1 runtime correctness打开 ETE 时仍应保留现有 fail-fast特别是 draft/EAGLE、logprob、hidden capture、compute padding 路径。
3. 如果 ETE 中 `KVCapacityWaitError` 频繁出现,下一步应把 owner-lane free pages/evictable pages 的 batch precheck 前移到 scheduler而不是引入 all-reduce。
## 19. 2026-06-04 L2 host write admission batch 化
### 背景
打开 scheduler bs>1 gate 后CP HiCache write-through 的 radix node / host reservation 仍然必须保持 per request 独立;但 host 侧容量 admission 如果仍按 request 串行执行,会在同一个 prefill batch 内反复触发 L2 host free-room eviction
- correctness 没问题,因为 `reserve_write_cp()` 最终会消费真实 host allocator slot
- 性能和碎片化较差,因为每个 request 都可能单独计算一次 `required + free_room_target - available`,在 host 接近满载时容易多次小额 eviction
- 这与 L1/L2 free-room 的目标相反:应尽量一次 evict 到目标余量,降低 eviction 频率并给后续 allocate 更连续的空间。
### 实现原则
只把 **host write admission / eviction planning** 聚合到 batch level不合并 request 生命周期:
- `CpHiCacheNodeMetadata`、host slot reservation、draft host slot、radix attach、rollback、ack 仍 per request
- batch admission 只聚合多个 request 的 `required_by_owner`,用一次 `_cp_build_write_admission_from_required()` 计算 host free-room deficit
- 如果 batch admission 需要 eviction则只在 forward 前触发一次 `_evict_cp_host_for_write_admission()`
- admission 成功后,每个 request 继续调用 `reserve_write_cp()` 拿自己的 host slots但跳过重复的 per-request initial admission
- 如果 batch admission 无法满足 deficit会显式打 `[CP_HICACHE_FALLBACK][prepare_write_backups_batch_admission_failed]`,再回到 per-request reservation 路径,不 silent fallback
- 不新增 collective不改变 CP owner-lane 分布,不改变 TAI kernel 接口。
### 完成状态
代码路径:
- `Scheduler._prepare_hicache_write_backups_before_forward()` 优先调用 `prepare_write_backups_for_reqs(batch.reqs)`
- `SessionAwareCache.prepare_write_backups_for_reqs()` 透传到 inner cache缺失 batch API 时才逐 req fallback
- `HiRadixCache.prepare_write_backups_for_reqs()`
- 先构造每个 request 的 `CpWriteBackupCandidate`
- 聚合所有 candidate 的 `_cp_required_host_tokens_by_rank(kv_indices)`
- 对聚合后的 `required_by_owner` 做一次 host write admission / eviction
- 再逐 request 做实际 reservation 和 per-layer backup registration。
新增测试:
- `test_cp_host_write_batch_admission_uses_aggregate_required`
- 验证 batch admission 对聚合 required 使用 free-room 公式。
- `test_prepare_write_backups_for_reqs_runs_one_batch_admission`
- 验证两个 request 只触发一次 batch admission / eviction最终仍各自得到 prepared backup并提交两个独立 reservation。
验证:
```text
python -m py_compile \
python/sglang/srt/mem_cache/hiradix_cache.py \
python/sglang/srt/mem_cache/session_aware_cache.py \
python/sglang/srt/managers/scheduler.py \
test/registered/unit/mem_cache/test_cp_hicache_metadata.py
PYTHONPATH=python python -m pytest -q \
test/registered/unit/mem_cache/test_cp_hicache_metadata.py::TestCpHiCacheFreeRoom
=> 8 passed, 3 warnings
PYTHONPATH=python python -m pytest -q \
test/registered/unit/mem_cache/test_cp_hicache_metadata.py
=> 117 passed, 5 warnings
```