laoyao0822 2c94b8de23 Ground HiCache backup overlap policy in measured tails ...

The per-layer backup path did not explain end-to-end performance by inspection alone, so add a focused benchmark that compares all-layer tail backup against per-layer forward-overlap backup under the same page_first_direct layout. The benchmark reports total latency, over-forward-only latency, submit overhead, and winner so short-extend fallback decisions are based on measured cost rather than request-size intuition.

Constraint: CUDA benchmark execution is remote-only; this local commit only records the benchmark driver.

Rejected: Adding more online logs for this question | the user asked for benchmark evidence and logs would perturb scheduler behavior.

Confidence: high

Scope-risk: narrow

Directive: Do not infer production policy from per_extra alone; per-layer CPU submission can be hidden inside the forward timing interval, so total latency is the safer comparison metric.

Tested: Local py_compile benchmark/hicache/bench_cp_hicache_backup_overlap.py.

Tested: Remote benchmark logs generated under /mnt/beegfs/cjy/cp_hicache_backup_overlap_bench_20260527_*.log.

Not-tested: Full SGLang test suite; benchmark is standalone and CUDA execution was remote-only.

2026-05-28 03:43:05 +08:00

14 KiB

Raw Blame History

View Raw