Batch CP HiCache host admission before write reservation
CP shared-KV prefill batching can present multiple requests to HiCache write-through in one forward. Keeping host free-room admission per request causes repeated small L2 evictions and defeats the free-room policy, while request metadata and radix attachment still need per-request ownership.\n\nThis change adds a batch prepare path that aggregates required host tokens by CP owner lane, performs one host admission/eviction step for the batch, and then reserves/submits each request independently. SessionAwareCache and the scheduler prefer the batch API when available.\n\nConstraint: Radix nodes, host slots, draft slots, rollback, and ack semantics remain per request.\nRejected: Merge reservations or radix nodes across requests | would complicate rollback and writing_check ack ownership.\nRejected: Add collective synchronization for host admission | local owner-lane logic already provides deterministic capacity accounting.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not reintroduce per-request host free-room eviction in the scheduler path without profiling host-full workloads.\nTested: local py_compile for touched Python files\nTested: remote g0034 docker PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 117 passed, 5 warnings\nNot-tested: full ETE bs>1 HiCache run under production traffic
This commit is contained in:
@@ -935,3 +935,61 @@ scheduler admission 不做 bf16/fp8 字节级估算。原因是实际 KV pool
|
||||
1. `cp_shared_kv_prefill_max_total_extend_tokens` 首版使用 page-aligned extend 累计,和 allocator 的最小 page 单位一致;日志/指标里如果要展示用户侧 token,需要另加 valid-token 统计。
|
||||
2. 该 gate 只控制 scheduler 组 batch,不解决所有 bs>1 runtime correctness;打开 ETE 时仍应保留现有 fail-fast,特别是 draft/EAGLE、logprob、hidden capture、compute padding 路径。
|
||||
3. 如果 ETE 中 `KVCapacityWaitError` 频繁出现,下一步应把 owner-lane free pages/evictable pages 的 batch precheck 前移到 scheduler,而不是引入 all-reduce。
|
||||
|
||||
## 19. 2026-06-04 L2 host write admission batch 化
|
||||
|
||||
### 背景
|
||||
|
||||
打开 scheduler bs>1 gate 后,CP HiCache write-through 的 radix node / host reservation 仍然必须保持 per request 独立;但 host 侧容量 admission 如果仍按 request 串行执行,会在同一个 prefill batch 内反复触发 L2 host free-room eviction:
|
||||
|
||||
- correctness 没问题,因为 `reserve_write_cp()` 最终会消费真实 host allocator slot;
|
||||
- 性能和碎片化较差,因为每个 request 都可能单独计算一次 `required + free_room_target - available`,在 host 接近满载时容易多次小额 eviction;
|
||||
- 这与 L1/L2 free-room 的目标相反:应尽量一次 evict 到目标余量,降低 eviction 频率并给后续 allocate 更连续的空间。
|
||||
|
||||
### 实现原则
|
||||
|
||||
只把 **host write admission / eviction planning** 聚合到 batch level,不合并 request 生命周期:
|
||||
|
||||
- `CpHiCacheNodeMetadata`、host slot reservation、draft host slot、radix attach、rollback、ack 仍 per request;
|
||||
- batch admission 只聚合多个 request 的 `required_by_owner`,用一次 `_cp_build_write_admission_from_required()` 计算 host free-room deficit;
|
||||
- 如果 batch admission 需要 eviction,则只在 forward 前触发一次 `_evict_cp_host_for_write_admission()`;
|
||||
- admission 成功后,每个 request 继续调用 `reserve_write_cp()` 拿自己的 host slots,但跳过重复的 per-request initial admission;
|
||||
- 如果 batch admission 无法满足 deficit,会显式打 `[CP_HICACHE_FALLBACK][prepare_write_backups_batch_admission_failed]`,再回到 per-request reservation 路径,不 silent fallback;
|
||||
- 不新增 collective,不改变 CP owner-lane 分布,不改变 TAI kernel 接口。
|
||||
|
||||
### 完成状态
|
||||
|
||||
代码路径:
|
||||
|
||||
- `Scheduler._prepare_hicache_write_backups_before_forward()` 优先调用 `prepare_write_backups_for_reqs(batch.reqs)`;
|
||||
- `SessionAwareCache.prepare_write_backups_for_reqs()` 透传到 inner cache,缺失 batch API 时才逐 req fallback;
|
||||
- `HiRadixCache.prepare_write_backups_for_reqs()`:
|
||||
- 先构造每个 request 的 `CpWriteBackupCandidate`;
|
||||
- 聚合所有 candidate 的 `_cp_required_host_tokens_by_rank(kv_indices)`;
|
||||
- 对聚合后的 `required_by_owner` 做一次 host write admission / eviction;
|
||||
- 再逐 request 做实际 reservation 和 per-layer backup registration。
|
||||
|
||||
新增测试:
|
||||
|
||||
- `test_cp_host_write_batch_admission_uses_aggregate_required`
|
||||
- 验证 batch admission 对聚合 required 使用 free-room 公式。
|
||||
- `test_prepare_write_backups_for_reqs_runs_one_batch_admission`
|
||||
- 验证两个 request 只触发一次 batch admission / eviction,最终仍各自得到 prepared backup,并提交两个独立 reservation。
|
||||
|
||||
验证:
|
||||
|
||||
```text
|
||||
python -m py_compile \
|
||||
python/sglang/srt/mem_cache/hiradix_cache.py \
|
||||
python/sglang/srt/mem_cache/session_aware_cache.py \
|
||||
python/sglang/srt/managers/scheduler.py \
|
||||
test/registered/unit/mem_cache/test_cp_hicache_metadata.py
|
||||
|
||||
PYTHONPATH=python python -m pytest -q \
|
||||
test/registered/unit/mem_cache/test_cp_hicache_metadata.py::TestCpHiCacheFreeRoom
|
||||
=> 8 passed, 3 warnings
|
||||
|
||||
PYTHONPATH=python python -m pytest -q \
|
||||
test/registered/unit/mem_cache/test_cp_hicache_metadata.py
|
||||
=> 117 passed, 5 warnings
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user