Remove full-cache scans from CP owner-lane allocation

The CP shared-KV allocator was still doing total-cache-sized CPU work in the scheduler hot path.  That cannot be hidden by GPU overlap, so owner-lane allocation now maintains per-owner free/release buckets and consumes request-sized prefixes instead of rebuilding masks over the full free-page tensor on each request.\n\nThe benchmark was extended to isolate L1 stats, selection, and allocation costs, and the CPU layout tests now install a complete sgl_kernel stub before importing SGLang helpers so remote unit collection does not abort in native extension loading.\n\nConstraint: Allocator CPU work blocks scheduler progress and cannot overlap with GPU forward execution.\nConstraint: CPU unit tests must not load native sgl_kernel on remote images where the loader can SIGABRT.\nRejected: Keep contiguous-run search over full free_pages | still scales with cache capacity and measured multi-ms overhead.\nRejected: Treat remote collection abort as an environment-only issue | it prevented allocator regression coverage and was fixable with a test-local stub.\nConfidence: high\nScope-risk: moderate\nDirective: CP owner-lane allocation is bucket-based; do not reintroduce full free_pages scans on the hot path without benchmark evidence.\nTested: Local py_compile for touched files\nTested: Local benchmark unit test, 6 passed\nTested: Remote benchmark unit test, 6 passed\nTested: Remote test_alloc_pages_with_owners.py, 10 passed\nTested: Remote test_cp_shared_kv_layout.py, 27 passed\nTested: Remote production allocator microbench shows select/alloc p50 reduced from ms-scale to sub-ms scale\nNot-tested: Full ETE traffic run after allocator bucket change

docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md

@@ -5150,3 +5150,164 @@ Remaining validation gap:
 - Requires a fresh ETE run with chunked prefill enabled to verify that the
   fallback storm disappears in production traffic.  An already-running remote
   process will not pick up this change until restarted.
+
+### C121 — 2026-06-02 CPU overhead must be measured by microbench, not inferred from logs
+
+Finding:
+
+- Runtime logs can show fallback storms and fatal paths, but they cannot rank normal-path CPU overhead.  The hot paths here are often successful allocator/control-path calls that produce no warning log.
+- The CP shared-KV L1 owner-lane allocator had two visible metadata costs:
+  - `compute_owner_lane_stats()` scanned the full free-page tensor once per CP owner and performed one `.item()` synchronization per owner.
+  - `_select_compute_owner_pages()` recomputed owner masks per owner and built prefix selections via full-tensor `cumsum`, then optionally searched contiguous owner-lane runs.
+- Host/L2 `alloc_contiguous_preferred()` is also expensive on fragmented metadata because it validates page contiguity over the full free-slot tensor and searches for later runs.  A local 210GB-equivalent metadata benchmark showed fragmented/random contiguous-preferred allocation at roughly 8–11 ms p50 versus FIFO at sub-ms scale for the same request sizes.
+
+Correction implemented:
+
+- Extended `benchmark/hicache/bench_cp_hicache_allocator_overhead.py` with L1 operation breakdowns:
+  - `stats`
+  - `free_room_stats`
+  - `select_only`
+  - `alloc_pages`
+- Added a dependency-light standalone CP shared-paged allocator model so the CPU metadata shape can be measured locally without importing the full SGLang runtime stack.
+- Optimized the production CP owner-lane stats path from per-owner mask+sum scans to one `torch.bincount()` over page owners.
+- Optimized owner-page selection by computing free/release page owners once per allocation and replacing the prefix `cumsum` selection with `nonzero()[:required_count]` prefix selection.
+- Replaced contiguous-run detection from `run_edges.unfold(...).all(dim=1)` with a cumulative-sum sliding-window test.  This preserves the same contiguous-run contract but avoids O(num_free_pages * required_pages) metadata work for large requests.
+
+Current evidence:
+
+- Local CPU benchmark after the stats/path cleanup still shows selection/allocation as the dominant cost.  Example focused run at `physical_pages=32768`, `cp_size=8`, random/zigzag owners:
+  - `stats`: about 0.9–1.4 ms p50.
+  - `select_only`: about 8–12 ms p50 for the focused random/fragmented zigzag cases.
+  - `alloc_pages`: about 8–12 ms p50 for the same cases, request sizes 64–512 pages.
+- Therefore the first optimization only reduces part of the overhead; the main remaining cost is still owner-lane page selection, especially contiguous-preferred search and full-size mask materialization.
+
+Verification:
+
+- Local:
+  - `python -m py_compile python/sglang/srt/mem_cache/allocator.py benchmark/hicache/bench_cp_hicache_allocator_overhead.py test/registered/unit/benchmark/test_cp_hicache_allocator_bench.py`
+  - `PYTHONPATH=. python -m pytest -q test/registered/unit/benchmark/test_cp_hicache_allocator_bench.py` → `6 passed, 1 warning`.
+- Remote `g0034` container:
+  - benchmark unit subset: `6 passed, 1 warning`.
+
+Known verification gap:
+
+- Remote `test_cp_shared_kv_layout.py` collection aborted while importing `sgl_kernel` native ops in the container, before reaching the allocator tests.  This is an environment/native import failure during test collection, not evidence of allocator logic failure.
+- Need a clean remote production-allocator test path or a container state where `sgl_kernel` imports safely before claiming full production allocator verification.
+
+Next target:
+
+- Replace full-tensor owner-lane selection with a lower-overhead data structure or batched selector.  The viable directions are:
+  1. Maintain per-owner free-page queues/counters incrementally.
+  2. Use a bounded contiguous-search policy and skip expensive run search for small requests or highly fragmented pools.
+  3. Add a fused selector kernel if GPU-side selection remains acceptable, but avoid increasing synchronization frequency.
+
+C121 additional remote standalone benchmark evidence:
+
+- `g0034` container, dependency-light standalone allocator model, `physical_pages=32768`, `cp_size=8`, zigzag owners:
+  - random, 64 pages: `stats` p50 0.87 ms, `select_only` p50 6.55 ms, `alloc_pages` p50 7.29 ms.
+  - owner-fragmented later-run, 64 pages: `stats` p50 0.43 ms, `select_only` p50 6.13 ms, `alloc_pages` p50 6.80 ms.
+  - random, 512 pages: `stats` p50 0.90 ms, `select_only` p50 7.46 ms, `alloc_pages` p50 7.91 ms.
+  - owner-fragmented later-run, 512 pages: `stats` p50 0.45 ms, `select_only` p50 6.89 ms, `alloc_pages` p50 7.16 ms.
+- This reinforces that after the cheap cleanup, stats is no longer the main bottleneck; selector/allocation metadata still costs multi-ms and is the next CPU-overhead target.
+
+### C122 — 2026-06-02 L1 owner-lane allocation must remove pure-CPU full scans
+
+Finding:
+
+- A remaining ~7–10 ms allocator/control-path cost is still too high because it
+  is pure CPU metadata work.  Unlike D2H/H2D backup/load kernels, this work
+  cannot overlap with GPU forward progress once the scheduler is blocked waiting
+  for page allocation.
+- The C121 cleanup made stats cheaper, but selection still scanned or
+  materialized full free-page tensors on every owner-lane allocation.
+- This shape scales with total cache capacity, not request size.  With a 150–220
+  GB HiCache/L1 metadata scale, even a successful hot-path allocation becomes a
+  scheduler stall.
+
+Correction in progress:
+
+- Make `CPSharedPagedTokenToKVPoolAllocator` keep per-owner free/release page
+  buckets as allocator state instead of deriving owner buckets by scanning
+  `free_pages` for every allocation.
+- Keep allocation itself request-sized: count required pages per owner, take the
+  needed prefix from that owner bucket, and consume bucket prefixes only after
+  all lanes are known satisfiable.
+- Preserve the public `free_pages` / `release_pages` tensor interface through
+  lazy materialized caches for tests and legacy paths, but avoid materializing it
+  in the CP owner-lane fast path.
+- Sort within each owner bucket when pages are inserted/restored.  This changes
+  the CP owner allocator from global FIFO semantics to owner-lane contiguous
+  semantics, which is more aligned with the RDMA/H2D/D2H goal: each owner lane
+  should prefer physically consecutive pages without a per-allocation run scan.
+
+Risk / contract note:
+
+- Code that only depends on owner-correct page allocation is unaffected.
+- Code or tests that implicitly relied on exact global `free_pages` order after a
+  CP owner-lane allocation must be updated; global FIFO ordering is not the
+  intended CP shared-KV owner-lane contract.
+
+C122 validation update:
+
+- Remote `g0034` production allocator microbench with `--stub-sgl-kernel`,
+  `physical_pages=32768`, `cp_size=8`, zigzag owners:
+  - random, 64 pages: `stats` p50 17 us, `select_only` p50 173 us,
+    `alloc_pages` p50 527 us.
+  - owner-fragmented later-run, 64 pages: `stats` p50 13 us,
+    `select_only` p50 166 us, `alloc_pages` p50 235 us.
+  - random, 512 pages: `stats` p50 27 us, `select_only` p50 284 us,
+    `alloc_pages` p50 638 us.
+  - owner-fragmented later-run, 512 pages: `stats` p50 32 us,
+    `select_only` p50 266 us, `alloc_pages` p50 476 us.
+- This confirms the allocator hot path is no longer full-cache-scan shaped for
+  this benchmark: the previous 6–8 ms selector/allocation p50 is reduced to
+  sub-ms p50 while preserving owner-lane correctness and contiguous lane
+  selections.
+- Remaining CPU cost is now mostly request-sized construction (`page_compute_owners`
+  grouping, output page tensor fill, and token-loc expansion), not total-cache
+  sized metadata scans.
+
+### C123 — 2026-06-02 CPU allocator tests must not load native sgl_kernel during collection
+
+Finding:
+
+- Remote `test_cp_shared_kv_layout.py` aborted during pytest collection inside
+  `sgl_kernel/load_utils.py::_load_architecture_specific_ops` before any test
+  logic ran.
+- This is not an allocator correctness failure.  The test imported
+  `sglang.test.test_utils`, which imports `sglang.srt.utils.common`; that module
+  imports `sgl_kernel` to probe AMX availability.  On the remote image the native
+  loader aborts the process instead of raising a catchable Python exception.
+- A `try: import sgl_kernel` fallback is insufficient for this environment
+  because SIGABRT bypasses Python exception handling.
+
+Correction:
+
+- Install a minimal `sgl_kernel`, `sgl_kernel.kvcacheio`, and
+  `sgl_kernel.quantization` stub at the top of CPU-only allocator/layout tests
+  before importing any `sglang` helper module.
+- Keep Torch custom-op schema registration for the operators referenced by the
+  imported SGLang code, but do not load the native extension during collection.
+
+Scope:
+
+- This applies only to CPU unit tests.  It does not change production runtime
+  import behavior and does not mask native kernel problems in CUDA/ETE tests.
+
+C123 validation update:
+
+- After avoiding the native import abort, collection reached Python test logic but
+  failed because the CPU stub had not defined the fp8 quantization custom-op
+  schemas used by `fp8_kernel.py` fake registrations.
+- The test stub now defines the fp8 quantization and fp8 GEMM schemas before any
+  `sglang` import, matching the CPU-only pattern used by the heavier HiCache
+  metadata tests.
+
+C123 full-suite update:
+
+- The full `test_cp_shared_kv_layout.py` file then reached scheduler rollback
+  tests and failed because `memory_pool_host.py` imports named transfer helpers
+  from `sgl_kernel.kvcacheio`.
+- The CPU stub now also gives `sgl_kernel.kvcacheio` a module-level
+  `__getattr__`, so named imports resolve to inert functions without importing
+  the native extension.