|
|
|
@@ -0,0 +1,847 @@
|
|
|
|
|
# NSA Prefill CP Page-Aligned Cache Contract
|
|
|
|
|
|
|
|
|
|
This note records the contract we need before continuing MLA/index partial
|
|
|
|
|
reuse under CP shared KV + HiCache + EAGLE/MTP.
|
|
|
|
|
|
|
|
|
|
The immediate trigger was a production hang where the prefill process stopped
|
|
|
|
|
after target-layer MLA partial current reuse:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
/mnt/beegfs/cjy/sglang_cp_hicache_20260528_161408.log
|
|
|
|
|
|
|
|
|
|
layer=2
|
|
|
|
|
prefix_lens=[13184]
|
|
|
|
|
extend_lens=[37024]
|
|
|
|
|
page_size=64
|
|
|
|
|
prefix_pages=206
|
|
|
|
|
suffix_slots=579
|
|
|
|
|
current_rows=37024
|
|
|
|
|
kv_rows=87328
|
|
|
|
|
used_prefetch=True
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`suffix_slots * page_size = 579 * 64 = 37056`, but only `37024` current
|
|
|
|
|
rows were appended. The fast path consumed prefetched prefix pages and appended
|
|
|
|
|
only real current rows, so the last suffix page had `32` rows of page slack with
|
|
|
|
|
no explicit valid/invalid contract.
|
|
|
|
|
|
|
|
|
|
The target design is **not** to fall back for these short or unaligned suffixes.
|
|
|
|
|
Instead, all physical cache paths should share one page-aligned contract while
|
|
|
|
|
logical token semantics stay unpadded.
|
|
|
|
|
|
|
|
|
|
## Contract
|
|
|
|
|
|
|
|
|
|
Use two lengths everywhere the path crosses a logical/physical boundary:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
valid_tokens = real request/cache tokens
|
|
|
|
|
padded_pages = ceil(valid_tokens / page_size)
|
|
|
|
|
padded_tokens = padded_pages * page_size
|
|
|
|
|
padding_tokens = padded_tokens - valid_tokens
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Logical layer: always valid-token based
|
|
|
|
|
|
|
|
|
|
The following must continue to use `valid_tokens`:
|
|
|
|
|
|
|
|
|
|
- request token length,
|
|
|
|
|
- `seq_lens_cpu`,
|
|
|
|
|
- `extend_seq_lens_cpu`,
|
|
|
|
|
- radix tree key/logical node length,
|
|
|
|
|
- cache-hit logical length exposed to scheduler,
|
|
|
|
|
- decode position and max-context checks,
|
|
|
|
|
- EAGLE/MTP verification token count,
|
|
|
|
|
- attention-visible sequence length.
|
|
|
|
|
|
|
|
|
|
Padding tokens must not become request tokens.
|
|
|
|
|
|
|
|
|
|
### Physical layer: always page-aligned
|
|
|
|
|
|
|
|
|
|
The following should use `padded_pages` / `padded_tokens`:
|
|
|
|
|
|
|
|
|
|
- GPU KV pool allocation and owner-lane capacity,
|
|
|
|
|
- draft KV mirror allocation,
|
|
|
|
|
- HiCache host reservation,
|
|
|
|
|
- backup/load/evict accounting,
|
|
|
|
|
- disaggregated target/draft transfer page lists,
|
|
|
|
|
- CP shared-KV MLA/index dense materialization buffers,
|
|
|
|
|
- CP owner-lane capacity planning and snapshots.
|
|
|
|
|
|
|
|
|
|
Padding rows may exist in temporary dense buffers or physical pages, but they
|
|
|
|
|
must be invisible to attention and radix semantics.
|
|
|
|
|
|
|
|
|
|
### Visibility rule
|
|
|
|
|
|
|
|
|
|
Every remap path must preserve this invariant:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
logical loc maps to a dense row only if it is a real valid token.
|
|
|
|
|
padding/slack locs map to -1.
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Padding KV rows can be zero-filled. Correctness comes from masking/remapping
|
|
|
|
|
them out, not from their value.
|
|
|
|
|
|
|
|
|
|
### Kernel ownership
|
|
|
|
|
|
|
|
|
|
Production hot paths should use `tai-kernel` for page-padded compose/remap and
|
|
|
|
|
copy-style operations. Python/Torch implementations are acceptable as CPU unit
|
|
|
|
|
test references and explicit warning fallbacks only; they should not be the
|
|
|
|
|
normal CUDA runtime path.
|
|
|
|
|
|
|
|
|
|
For MLA partial-current reuse, the intended runtime primitive is:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
preallocated slot-dense KV pages
|
|
|
|
|
+ current suffix KV copied into its slot-dense rows
|
|
|
|
|
+ zero padding rows left invisible
|
|
|
|
|
-> same dense KV buffer + mixed loc remap + current mask
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
This should be one `tai_kernel.nsa_prefill.cp_shared_kv_materialize` primitive,
|
|
|
|
|
not a sequence of `torch.sort/searchsorted/cat/zeros` operations in the
|
|
|
|
|
attention hot path. If the TAI primitive is unavailable, SGLang must emit a
|
|
|
|
|
visible warning fallback rather than silently taking the slow path.
|
|
|
|
|
|
|
|
|
|
The TAI primitive should extend the existing materialize/remap kernel family:
|
|
|
|
|
|
|
|
|
|
- copy each valid `current_locs[i]` row into `page_inverse[page] * page_size + offset`;
|
|
|
|
|
- build a current-page inverse from page-grouped current locs;
|
|
|
|
|
- keep real current locs visible by exact loc check;
|
|
|
|
|
- map non-current rows inside current pages to `-1`.
|
|
|
|
|
|
|
|
|
|
This avoids the previous compact-current append path where tail-page slack had
|
|
|
|
|
slot-dense locs but no valid current KV row.
|
|
|
|
|
|
|
|
|
|
## Root-cause record: unsafe partial-current prefetch
|
|
|
|
|
|
|
|
|
|
The current MLA partial-current prefetch path has a narrower contract than the
|
|
|
|
|
page-level metadata it consumes.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `cp_shared_kv_prefetch.py::consume_prefix_with_current()` says the prefetched
|
|
|
|
|
prefix is already materialized while current/suffix pages are not copied from
|
|
|
|
|
the shared pool; it then appends `current_kv_cache` directly.
|
|
|
|
|
- `cp_shared_kv_runtime.py::merge_materialized_and_current_kv()` only remaps
|
|
|
|
|
entries present in `current_locs`; all non-current entries keep the existing
|
|
|
|
|
`materialized_dense_locs`.
|
|
|
|
|
- `nsa_backend.py` calls `consume_prefix_with_current()` whenever an MLA
|
|
|
|
|
prefetcher exists on a mixed prefix/current batch, before falling back to the
|
|
|
|
|
older partial materialize path.
|
|
|
|
|
|
|
|
|
|
That is safe only when the page-level suffix represented by the page table is
|
|
|
|
|
exactly covered by `current_locs`. It is not safe when:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
ceil(extend_len / page_size) * page_size > extend_len
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The older non-prefetch partial path is safer because it materializes all
|
|
|
|
|
non-current logical locations and only substitutes rows that match
|
|
|
|
|
`current_locs`.
|
|
|
|
|
|
|
|
|
|
The fix should not be a permanent fallback. The intended fix is to make the
|
|
|
|
|
prefetch compose path create a page-padded current suffix buffer and map suffix
|
|
|
|
|
slack to `-1`.
|
|
|
|
|
|
|
|
|
|
## Existing evidence in the codebase
|
|
|
|
|
|
|
|
|
|
### Page-aligned split already treats tail as a page unit
|
|
|
|
|
|
|
|
|
|
`build_page_aligned_in_seq_split_list()` computes:
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
full_pages = extend_len // page_size
|
|
|
|
|
tail_tokens = extend_len % page_size
|
|
|
|
|
num_page_units = full_pages + (1 if tail_tokens > 0 else 0)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The Phase 4 doc explicitly states that the tail unit is not split across CP
|
|
|
|
|
segments. However, the current `split_list` still sums to `extend_len`, not
|
|
|
|
|
`padded_tokens`. That is correct for compute tokens, but it means later cache
|
|
|
|
|
paths must not assume `split_list` size equals physical page coverage.
|
|
|
|
|
|
|
|
|
|
### HiCache metadata currently requires page-aligned logical_len
|
|
|
|
|
|
|
|
|
|
`CpHiCacheNodeMetadata.__post_init__()` rejects `logical_len % page_size != 0`,
|
|
|
|
|
and `reserve_write_cp()` rejects non-page-aligned `device_indices`.
|
|
|
|
|
|
|
|
|
|
This is already a physical-cache assumption. If logical radix length remains
|
|
|
|
|
valid-token based, the metadata needs a clearer name/split:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
valid_len # radix-visible logical length
|
|
|
|
|
padded_len # physical cache/host metadata length
|
|
|
|
|
page_owners # length padded_len / page_size
|
|
|
|
|
owned_positions # local positions inside padded_len
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Without this split, node split/load checks can accidentally force logical cache
|
|
|
|
|
semantics to be page-aligned.
|
|
|
|
|
|
|
|
|
|
### Owner-lane accounting is already page-derived
|
|
|
|
|
|
|
|
|
|
`HiRadixCache._cp_owner_token_counts()` counts `page_size` tokens per owner page.
|
|
|
|
|
The no-collective capacity plan already describes a per-owner page ledger. This
|
|
|
|
|
means the capacity side is structurally ready for page-based accounting, but all
|
|
|
|
|
inputs to the ledger must be padded-page based. A valid-token scalar is not
|
|
|
|
|
enough for CP owner-lane admission.
|
|
|
|
|
|
|
|
|
|
`cp_shared_kv_compute_owner.build_in_seq_page_compute_owners()` should therefore
|
|
|
|
|
return owners for every physical page unit, including short chunks where some
|
|
|
|
|
zigzag segments own zero pages. Falling back only because `num_pages < 2 *
|
|
|
|
|
cp_size` breaks the page-owner lane contract and can force later materialize or
|
|
|
|
|
HiCache paths back to token-balanced behavior.
|
|
|
|
|
|
|
|
|
|
This owner assignment remains page-count based; `out_cache_loc` remains
|
|
|
|
|
valid-token based. `alloc_extend_compute_owner()` uses the owner list to choose
|
|
|
|
|
whole physical pages, then `alloc_extend_naive()` returns exactly the valid
|
|
|
|
|
extend token locs. The padded tail rows exist because the selected logical page
|
|
|
|
|
exists, not because fake token locs are appended to `out_cache_loc`.
|
|
|
|
|
|
|
|
|
|
### Load must replay page owners
|
|
|
|
|
|
|
|
|
|
`HiCacheController.load_cp()` reconstructs device allocation via
|
|
|
|
|
`alloc_pages_with_owners(page_owners)`. This confirms that load-back is already
|
|
|
|
|
page-pattern based and should stay physical-page based.
|
|
|
|
|
|
|
|
|
|
## Impact surface
|
|
|
|
|
|
|
|
|
|
### 1. CP split and metadata
|
|
|
|
|
|
|
|
|
|
Files:
|
|
|
|
|
|
|
|
|
|
- `python/sglang/srt/layers/attention/nsa/utils.py`
|
|
|
|
|
|
|
|
|
|
Needed changes:
|
|
|
|
|
|
|
|
|
|
- keep `split_list` and `actual_token_count` valid-token based;
|
|
|
|
|
- add explicit padded page metadata to `NSAContextParallelMetadata` or an
|
|
|
|
|
attached helper object;
|
|
|
|
|
- keep `segment_page_starts/ends` page-based, including tail page coverage;
|
|
|
|
|
- add a helper that returns `valid_tokens`, `padded_tokens`, `padded_pages`, and
|
|
|
|
|
`padding_tokens` for the current extend/cache-hit suffix.
|
|
|
|
|
|
|
|
|
|
Risk:
|
|
|
|
|
|
|
|
|
|
- changing `split_list` to padded tokens would leak fake tokens into compute and
|
|
|
|
|
logits paths. Do not do that.
|
|
|
|
|
|
|
|
|
|
### 2. MLA partial current reuse
|
|
|
|
|
|
|
|
|
|
Files:
|
|
|
|
|
|
|
|
|
|
- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py`
|
|
|
|
|
- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_prefetch.py`
|
|
|
|
|
- `python/sglang/srt/layers/attention/nsa_backend.py`
|
|
|
|
|
|
|
|
|
|
Needed changes:
|
|
|
|
|
|
|
|
|
|
- use the existing page-slot dense buffer for mixed prefix/current reuse;
|
|
|
|
|
- copy valid current suffix rows into their slot-dense page rows with
|
|
|
|
|
`tai-kernel` in the CUDA path;
|
|
|
|
|
- keep padding rows in the tail suffix page physically present but zero/unused;
|
|
|
|
|
- remap valid current locs to their slot-dense rows;
|
|
|
|
|
- map suffix slack/future locs to `-1`;
|
|
|
|
|
- keep existing full/current-only path behavior unchanged.
|
|
|
|
|
|
|
|
|
|
Short-term safe gate if this cannot be implemented atomically:
|
|
|
|
|
|
|
|
|
|
- WARNING fallback to the older partial materialize path when suffix slack is
|
|
|
|
|
detected. This is a stopgap only; the target contract is page-padded compose,
|
|
|
|
|
not fallback.
|
|
|
|
|
|
|
|
|
|
### 3. NSA index partial reuse
|
|
|
|
|
|
|
|
|
|
Files:
|
|
|
|
|
|
|
|
|
|
- `python/sglang/srt/layers/attention/nsa/nsa_indexer.py`
|
|
|
|
|
- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_prefetch.py`
|
|
|
|
|
|
|
|
|
|
Needed changes:
|
|
|
|
|
|
|
|
|
|
- apply the same valid/padded contract as MLA;
|
|
|
|
|
- index page metadata can be reused across layers, but returned dense page/table
|
|
|
|
|
remaps must not expose padding locs as valid;
|
|
|
|
|
- current index reuse should not assume that all locations in a suffix page are
|
|
|
|
|
current tokens.
|
|
|
|
|
|
|
|
|
|
Risk:
|
|
|
|
|
|
|
|
|
|
- index is page-first by nature, so it is easy to accidentally treat the whole
|
|
|
|
|
tail page as visible. Tests must include non-page-aligned suffixes.
|
|
|
|
|
|
|
|
|
|
### 4. HiCache host metadata and radix split
|
|
|
|
|
|
|
|
|
|
Files:
|
|
|
|
|
|
|
|
|
|
- `python/sglang/srt/mem_cache/hiradix_cache.py`
|
|
|
|
|
- `python/sglang/srt/managers/cache_controller.py`
|
|
|
|
|
|
|
|
|
|
Needed changes:
|
|
|
|
|
|
|
|
|
|
- distinguish radix-visible `valid_len` from physical `padded_len`;
|
|
|
|
|
- store `page_owners` for `padded_len / page_size` pages;
|
|
|
|
|
- keep split rules page-aware: a radix split inside an in-flight or host-backed
|
|
|
|
|
padded page must either split at a page boundary or keep the child pending
|
|
|
|
|
until the physical operation is committed;
|
|
|
|
|
- make `_node_host_len()` / host-hit accounting return valid length to the
|
|
|
|
|
scheduler but use padded length for load/evict/host free.
|
|
|
|
|
|
|
|
|
|
Risk:
|
|
|
|
|
|
|
|
|
|
- if `host_hit_length` becomes padded, scheduler may believe fake tokens were
|
|
|
|
|
cached. If physical metadata uses valid length, host reservation/load can
|
|
|
|
|
under-allocate the tail page.
|
|
|
|
|
|
|
|
|
|
### 5. Capacity and owner-lane admission
|
|
|
|
|
|
|
|
|
|
Files:
|
|
|
|
|
|
|
|
|
|
- `python/sglang/srt/mem_cache/hiradix_cache.py`
|
|
|
|
|
- `python/sglang/srt/mem_cache/allocator.py`
|
|
|
|
|
- `python/sglang/srt/managers/cache_controller.py`
|
|
|
|
|
|
|
|
|
|
Needed changes:
|
|
|
|
|
|
|
|
|
|
- all required capacity should be expressed as pages or per-owner padded tokens;
|
|
|
|
|
- owner-lane vectors should be derived from padded `page_owners`, not valid
|
|
|
|
|
token counts;
|
|
|
|
|
- target and draft reservation must use the same padded page pattern;
|
|
|
|
|
- no hot-path collective should be introduced to discover the mismatch.
|
|
|
|
|
|
|
|
|
|
Risk:
|
|
|
|
|
|
|
|
|
|
- if target uses padded capacity and draft uses valid capacity, draft can miss
|
|
|
|
|
the tail page while target succeeds, causing cache-hit accept-rate corruption.
|
|
|
|
|
|
|
|
|
|
### 6. Disaggregated transfer
|
|
|
|
|
|
|
|
|
|
Files:
|
|
|
|
|
|
|
|
|
|
- `python/sglang/srt/disaggregation/prefill.py`
|
|
|
|
|
- `python/sglang/srt/disaggregation/decode.py`
|
|
|
|
|
- `python/sglang/srt/disaggregation/mooncake/conn.py`
|
|
|
|
|
- `python/sglang/srt/disaggregation/nixl/conn.py`
|
|
|
|
|
|
|
|
|
|
Needed changes:
|
|
|
|
|
|
|
|
|
|
- transfer page lists remain page-based and may include the padded tail page;
|
|
|
|
|
- request metadata sent across disaggregation must keep valid token length;
|
|
|
|
|
- decode should not infer real prompt length from transferred padded page count;
|
|
|
|
|
- draft payload follows target pages, not independent draft token length.
|
|
|
|
|
|
|
|
|
|
Risk:
|
|
|
|
|
|
|
|
|
|
- page transfer count and logical request length diverge by up to
|
|
|
|
|
`page_size - 1` tokens. Any path that uses one as the other is wrong.
|
|
|
|
|
|
|
|
|
|
### 7. EAGLE/MTP draft KV
|
|
|
|
|
|
|
|
|
|
Files:
|
|
|
|
|
|
|
|
|
|
- `python/sglang/srt/speculative/eagle_worker.py`
|
|
|
|
|
- `python/sglang/srt/speculative/eagle_info.py`
|
|
|
|
|
- `python/sglang/srt/managers/cache_controller.py`
|
|
|
|
|
|
|
|
|
|
Needed changes:
|
|
|
|
|
|
|
|
|
|
- draft physical pages mirror target padded pages;
|
|
|
|
|
- draft valid length mirrors target valid length;
|
|
|
|
|
- draft cannot independently choose a shorter valid-only capacity path;
|
|
|
|
|
- draft cache-hit prefetch/current reuse must use the same visibility mask.
|
|
|
|
|
|
|
|
|
|
Risk:
|
|
|
|
|
|
|
|
|
|
- draft and target can both be physically page-aligned while disagreeing on
|
|
|
|
|
which rows are valid. That still breaks verification/accept rate.
|
|
|
|
|
|
|
|
|
|
## Missing work / correction ledger
|
|
|
|
|
|
|
|
|
|
This section is the working checklist for the current implementation. Each item
|
|
|
|
|
records what is still missing or what existing code needs to be corrected before
|
|
|
|
|
we can treat the page-aligned contract as complete.
|
|
|
|
|
|
|
|
|
|
### C1. Add an explicit page extent object
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- `build_page_aligned_in_seq_split_list()` computes `num_page_units` locally.
|
|
|
|
|
- Callers still pass raw lengths such as `extend_len`, `logical_len`, and
|
|
|
|
|
`required_slots` without naming whether they are valid-token or padded-page
|
|
|
|
|
quantities.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Add a small helper/data object, for example:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
PageAlignedCacheExtent(
|
|
|
|
|
valid_tokens,
|
|
|
|
|
padded_pages,
|
|
|
|
|
padded_tokens,
|
|
|
|
|
padding_tokens,
|
|
|
|
|
)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- Use it where the code crosses logical/physical boundaries.
|
|
|
|
|
- `valid_tokens=100, page_size=64` must produce `padded_pages=2` and
|
|
|
|
|
`padded_tokens=128`; it must not depend on `cp_size`.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Unit-test exact examples such as `0`, `64`, `100`, and `128` valid tokens.
|
|
|
|
|
- Add an assertion that `100` tokens with `page_size=64, cp_size=8` does not
|
|
|
|
|
become `1024` physical tokens.
|
|
|
|
|
|
|
|
|
|
### C2. Extend CP split metadata without padding `split_list`
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- `split_list` is correct because it sums to valid tokens.
|
|
|
|
|
- `PageAlignedInSeqSplitInfo` and `NSAContextParallelMetadata` expose only
|
|
|
|
|
`page_aligned`, `page_size`, `extend_prefix_len`, and segment page ranges.
|
|
|
|
|
- There is no explicit field carrying padded suffix coverage.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Keep `split_list`, `actual_token_count`, and query compute valid-token based.
|
|
|
|
|
- Add explicit metadata such as `extend_valid_tokens`, `extend_padded_pages`,
|
|
|
|
|
`extend_padded_tokens`, and `extend_padding_tokens`.
|
|
|
|
|
- Keep `segment_page_starts/ends` page-based and include the tail page coverage.
|
|
|
|
|
- Do not change `cp_split_and_rebuild_data()` to consume fake padding tokens.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- `extend_len=100, page_size=64, cp_size=8` should have:
|
|
|
|
|
- `sum(split_list) == 100`;
|
|
|
|
|
- `extend_padded_pages == 2`;
|
|
|
|
|
- `extend_padded_tokens == 128`;
|
|
|
|
|
- zero-token segments for ranks/zigzag segments without pages.
|
|
|
|
|
|
|
|
|
|
### C3. Remove short-suffix fallback completely, but only as metadata padding
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- The old `num_pages < cp_size` / `num_pages < 2 * cp_size` fallback was removed
|
|
|
|
|
from the split and compute-owner helpers.
|
|
|
|
|
- `should_use_replicated_compute_for_short_radix_hit()` now returns `False`.
|
|
|
|
|
- `can_cp_split()` still has generic CP minimum-token checks; those are compute
|
|
|
|
|
feasibility checks, not page-padding rules.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Keep the principle: no fallback solely because a suffix has fewer than
|
|
|
|
|
`cp_size` or `2 * cp_size` pages.
|
|
|
|
|
- Padding means "cover the partial tail page physically"; it does not mean
|
|
|
|
|
"round up to cp_size pages".
|
|
|
|
|
- If a truly tiny batch cannot run through CP compute, the reason must be a
|
|
|
|
|
clearly named compute limitation, not `too_short_for_page_aligned`.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- `extend_len=100, page_size=64, cp_size=8` should use two physical page units.
|
|
|
|
|
- `extend_len=3 pages, cp_size=8` should produce three compute owners, not
|
|
|
|
|
eight or sixteen.
|
|
|
|
|
|
|
|
|
|
### C4. MLA partial-current prefetch compose is partially corrected
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- `CpSharedKVMlaPrefetcher.consume_prefix_with_current()` now uses a page-slot
|
|
|
|
|
compose helper instead of compact-appending current KV rows.
|
|
|
|
|
- `fill_current_kv_page_slots_and_remap_locs()` copies valid current rows into
|
|
|
|
|
their dense page slots and masks non-current locs in current pages to `-1`.
|
|
|
|
|
- A TAI primitive exists for the CUDA hot path, with a visible warning fallback
|
|
|
|
|
to the Torch reference when unavailable.
|
|
|
|
|
|
|
|
|
|
Correction still needed:
|
|
|
|
|
|
|
|
|
|
- Verify that every MLA mixed prefix/current path uses the page-slot compose
|
|
|
|
|
helper when a dense slot buffer already exists.
|
|
|
|
|
- Audit the non-prefetch fallback in `nsa_backend.py`: it still calls
|
|
|
|
|
`merge_materialized_and_current_kv()`, which is the compact-current helper.
|
|
|
|
|
That path may be correct only when the materialized loc table has already
|
|
|
|
|
masked suffix slack. This must be proven by tests or changed to the same
|
|
|
|
|
page-slot helper.
|
|
|
|
|
- Keep `merge_materialized_and_current_kv()` documented as compact-materialize
|
|
|
|
|
only; do not silently use it for page-slot prefetch compose.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Existing MLA slack test covers a tail page where only part of the suffix page
|
|
|
|
|
is valid.
|
|
|
|
|
- Add a non-prefetch partial-current test with the same tail-page shape before
|
|
|
|
|
changing the fallback path.
|
|
|
|
|
- CUDA execution of the TAI primitive must be verified remotely; do not test
|
|
|
|
|
CUDA locally.
|
|
|
|
|
|
|
|
|
|
### C5. NSA index partial reuse still needs an equivalent padding contract
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- `CpSharedKVIndexPrefetcher.consume()` materializes suffix page slots into a
|
|
|
|
|
dense page buffer and returns dense pages/block tables.
|
|
|
|
|
- Index prefetch is page-first, so it can easily treat a whole tail page as
|
|
|
|
|
visible unless query/cache sequence lengths keep the tail rows masked.
|
|
|
|
|
- There is no dedicated unit test proving tail-page slack is invisible for NSA
|
|
|
|
|
index partial reuse.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Apply the same valid/padded distinction as MLA:
|
|
|
|
|
- page tables may include the padded tail page;
|
|
|
|
|
- valid sequence length must remain unpadded;
|
|
|
|
|
- any loc/table remap that can expose token rows must map padding rows to
|
|
|
|
|
invalid/sentinel entries or rely on a proven sequence-length mask.
|
|
|
|
|
- Current code inspection shows NSA index uses page ids plus explicit valid
|
|
|
|
|
sequence lengths: `GetKAndS` iterates `token_ids < seq_len`, and paged MQA
|
|
|
|
|
logits receive `seqlens_32` / `indexer_seq_lens`. Therefore tail-page slack
|
|
|
|
|
is not visible if those lengths remain valid-token based.
|
|
|
|
|
- The required immediate fix is a regression test that proves index prefetch keeps
|
|
|
|
|
the padded tail page in the dense page table while relying on valid
|
|
|
|
|
`indexer_seq_lens` to exclude slack. If a future path consumes index page
|
|
|
|
|
tables without an accompanying valid length, it must add an explicit
|
|
|
|
|
sentinel/remap or warning fallback before using the table.
|
|
|
|
|
- If index prefetch cannot prove this contract for a shape, emit a warning
|
|
|
|
|
fallback with a narrow reason instead of silently using a risky path.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Add an index partial reuse case with a non-page-aligned suffix:
|
|
|
|
|
- tail page included in dense pages;
|
|
|
|
|
- dense page table has only `ceil(valid_tokens / page_size)` physical pages,
|
|
|
|
|
not cp-size padding;
|
|
|
|
|
- valid sequence length remains the original unpadded token count so the
|
|
|
|
|
index kernels exclude tail slack.
|
|
|
|
|
|
|
|
|
|
### C6. HiCache metadata conflates logical and physical lengths
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- `CpHiCacheNodeMetadata.logical_len` is required to be page-aligned.
|
|
|
|
|
- `reserve_write_cp()` rejects non-page-aligned `device_indices`.
|
|
|
|
|
- `_build_cp_load_back_plan()` and `load_cp()` use `node.host_len` as both the
|
|
|
|
|
logical cache-hit length and physical load length.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Split the metadata into:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
valid_len # radix/scheduler-visible tokens
|
|
|
|
|
padded_len # physical host/device cache span
|
|
|
|
|
page_owners # length padded_len / page_size
|
|
|
|
|
owned_positions # positions inside padded_len
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- Scheduler-facing cache hit length must stay `valid_len`.
|
|
|
|
|
- Host reservation, backup, load, eviction, and owner-lane capacity must use
|
|
|
|
|
`padded_len`.
|
|
|
|
|
- `load_cp()` should validate allocation length against padded physical length,
|
|
|
|
|
while returning enough information for the radix/scheduler path to expose only
|
|
|
|
|
valid tokens.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Metadata with `valid_len=100, padded_len=128, page_size=64`.
|
|
|
|
|
- Host hit reports `100` to scheduler but reserves/loads `128` physical slots.
|
|
|
|
|
- `page_owners` length is `2`.
|
|
|
|
|
|
|
|
|
|
### C7. Radix page alignment currently floors partial tails
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- `page_align_keys()` truncates keys to a page boundary.
|
|
|
|
|
- `prepare_write_backup_for_req()` calls `page_align_keys()` before probing and
|
|
|
|
|
reserving CP HiCache backup.
|
|
|
|
|
- This can preserve correctness, but a short extend can be omitted from host
|
|
|
|
|
cache until it completes a full page. Repeated short suffixes can therefore
|
|
|
|
|
keep future hits on a fallback/partial path.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Do not append fake tokens to radix keys.
|
|
|
|
|
- Keep radix keys valid-token based, but attach physical padded metadata to the
|
|
|
|
|
node/backup record.
|
|
|
|
|
- If a radix split would land inside an in-flight padded physical page, either:
|
|
|
|
|
- delay/pending the split; or
|
|
|
|
|
- force the split to a page boundary and keep the valid-tail metadata attached
|
|
|
|
|
to the child.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Write a 100-token suffix and verify the radix-visible key length remains 100
|
|
|
|
|
while physical metadata covers 128 tokens.
|
|
|
|
|
- Split a node around the tail page and verify valid/padded lengths remain
|
|
|
|
|
consistent.
|
|
|
|
|
|
|
|
|
|
### C8. Owner-lane capacity must be padded-page based end to end
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- Compute-owner allocation returns one owner per physical page unit, including a
|
|
|
|
|
tail page.
|
|
|
|
|
- HiCache capacity planning counts `page_size` tokens per owner page.
|
|
|
|
|
- Some entry points still pass scalar valid lengths or page-aligned-only
|
|
|
|
|
`device_indices`.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Derive required capacity from padded `page_owners`, not valid token counts.
|
|
|
|
|
- Admission should use a per-owner vector built from the padded page list.
|
|
|
|
|
- Do not introduce an all-reduce/collective to discover capacity agreement; the
|
|
|
|
|
owner pattern should make the required vector deterministic.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- A 100-token suffix with owners `[0, 1]` should require one page on owner 0 and
|
|
|
|
|
one page on owner 1.
|
|
|
|
|
- Target and draft required vectors must match when draft HiCache is enabled.
|
|
|
|
|
|
|
|
|
|
### C9. Draft/EAGLE currently has safe fallbacks, not the final contract
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- Draft HiCache piggybacks target reservations/positions when
|
|
|
|
|
`draft_host_indices` are present.
|
|
|
|
|
- Draft partial-current reuse/prefetch has been disabled or pushed to older
|
|
|
|
|
materialization paths in some cases to avoid EAGLE hangs.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Draft physical pages must mirror target padded pages.
|
|
|
|
|
- Draft valid length must mirror target valid length.
|
|
|
|
|
- Draft cannot independently choose a valid-only capacity or prefetch path.
|
|
|
|
|
- Re-enable draft/EAGLE partial reuse only after the same page-slot visibility
|
|
|
|
|
mask is proven for draft.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Target/draft reservation with a non-page-aligned suffix uses identical
|
|
|
|
|
`owned_positions` and padded physical length.
|
|
|
|
|
- EAGLE one-layer path does not prefetch or consume an unmasked padded tail.
|
|
|
|
|
|
|
|
|
|
### C10. Disaggregated transfer has not been audited for valid/padded split
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- Transfer APIs are page-oriented in several paths, but request metadata remains
|
|
|
|
|
token-length oriented.
|
|
|
|
|
- There is no current checklist proving that decode never infers prompt length
|
|
|
|
|
from transferred padded page count.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Transfer page lists may include the padded tail page.
|
|
|
|
|
- Request/bootstrap metadata must carry valid token length separately.
|
|
|
|
|
- Decode-visible prompt length and max-context checks must use valid length.
|
|
|
|
|
- Draft transfer follows target page pattern.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Send a non-page-aligned prompt through disaggregation:
|
|
|
|
|
- transferred pages include the padded tail page;
|
|
|
|
|
- decode sees the original valid length;
|
|
|
|
|
- draft and target transfer page counts match when draft KV is enabled.
|
|
|
|
|
|
|
|
|
|
### C11. Fallback logging must be audited
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- MLA prefix misalignment now logs a warning fallback.
|
|
|
|
|
- Some prefetch misses use debug logs or normal fallback paths.
|
|
|
|
|
- Historical silent fallbacks made it hard to identify that async/per-layer or
|
|
|
|
|
page-aligned paths were not actually active.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Production-visible fallback from an intended hot path must emit
|
|
|
|
|
`[CP_SHARED_KV_FALLBACK]` or `[CP_HICACHE_FALLBACK]` at warning level with a
|
|
|
|
|
narrow reason.
|
|
|
|
|
- Debug-only logs are acceptable only for expected non-fallback states.
|
|
|
|
|
- Avoid per-layer spam for expected steady-state events; warn on path changes,
|
|
|
|
|
not every successful layer.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Unit-test representative fallback logs:
|
|
|
|
|
- MLA prefix not page-aligned;
|
|
|
|
|
- index prefetch consume miss after start layer;
|
|
|
|
|
- CP HiCache reservation/load capacity failure.
|
|
|
|
|
|
|
|
|
|
### C12. Verification gaps before claiming completion
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- Python syntax checks pass for the edited SGLang files.
|
|
|
|
|
- Local pytest collection can be blocked by missing optional dependencies such
|
|
|
|
|
as `pybase64`; this workspace also hit `orjson` during targeted
|
|
|
|
|
`test_nsa_cp_utils.py` collection, then `starlette` after temporary stubbing.
|
|
|
|
|
`test_cp_shared_kv_layout.py` collection also hit `transformers` via
|
|
|
|
|
`sglang.test.test_utils`.
|
|
|
|
|
- CUDA kernels must not be tested locally.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Use a temporary local test stub only for optional import blockers such as
|
|
|
|
|
`pybase64` / `orjson`; do not commit stubs.
|
|
|
|
|
- If dependency stubbing starts expanding into unrelated web/server packages,
|
|
|
|
|
stop the dependency-stub path and use isolated module loading or
|
|
|
|
|
`py_compile` for the targeted helper. Do not spend the implementation loop
|
|
|
|
|
repeatedly rediscovering local environment gaps.
|
|
|
|
|
- Run CPU/unit tests for extent/split/owner/runtime helpers.
|
|
|
|
|
- Run CUDA/TAI kernel verification only on the remote CUDA environment.
|
|
|
|
|
- Record remote log evidence for:
|
|
|
|
|
- TAI page-slot compose being used;
|
|
|
|
|
- no warning fallback on the expected hot path;
|
|
|
|
|
- cache-hit request completing without detokenizer hang.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Targeted CPU pytest for:
|
|
|
|
|
- `test_nsa_cp_utils.py`;
|
|
|
|
|
- `test_cp_shared_kv_layout.py`;
|
|
|
|
|
- `test_cp_shared_kv_runtime.py`.
|
|
|
|
|
- Remote CUDA benchmark/runtime for the TAI materialize primitive and full ETE
|
|
|
|
|
prefill run.
|
|
|
|
|
|
|
|
|
|
### C13. Remote source sync can expose mixed-version symbols
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- Remote container targeted new tests passed after scp sync.
|
|
|
|
|
- A wider remote run of:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
PYTHONPATH=python python -m pytest -q \
|
|
|
|
|
test/registered/unit/layers/test_nsa_cp_utils.py \
|
|
|
|
|
test/registered/unit/mem_cache/test_cp_shared_kv_layout.py \
|
|
|
|
|
test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
produced 102 passes and 19 failures.
|
|
|
|
|
- The dominant failure is an import mismatch:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
nsa_indexer.py / nsa_backend.py import
|
|
|
|
|
cp_shared_kv_mla_prefetch_should_trace_tiny_extend
|
|
|
|
|
from cp_shared_kv_runtime.py, but the synced runtime file does not export it.
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Treat this as a mixed-version/sync integrity problem before making logic
|
|
|
|
|
changes.
|
|
|
|
|
- Compare local and remote `cp_shared_kv_runtime.py`, `nsa_indexer.py`, and
|
|
|
|
|
`nsa_backend.py` symbol expectations.
|
|
|
|
|
- Either restore/export the expected helper or sync the matching caller files.
|
|
|
|
|
- Re-run the same remote wider suite after the symbol contract is repaired.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Import-level check:
|
|
|
|
|
`python - <<'PY' ... import cp_shared_kv_mla_prefetch_should_trace_tiny_extend`.
|
|
|
|
|
- Re-run the three-file remote pytest suite.
|
|
|
|
|
|
|
|
|
|
### C14. Wider remote unit suite still exposes non-page-padding regressions
|
|
|
|
|
|
|
|
|
|
Current state:
|
|
|
|
|
|
|
|
|
|
- After directly overwriting remote `nsa_indexer.py` and `nsa_backend.py` to
|
|
|
|
|
remove the mixed-version import mismatch, remote py_compile passed.
|
|
|
|
|
- The three-file remote unit suite improved to:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
114 passed, 7 failed, 5 warnings
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- Remaining failures are:
|
|
|
|
|
|
|
|
|
|
1. `test_local_out_cache_loc_logs_every_fallback_event` expects the old
|
|
|
|
|
fallback text `"CP shared KV direct-write fallback"`, while current code
|
|
|
|
|
emits the newer warning prefix
|
|
|
|
|
`[CP_SHARED_KV_FALLBACK][direct_write]`.
|
|
|
|
|
2. MLA and index prefetch stream tests expect prefix materialization to run
|
|
|
|
|
inside the prefetch stream context, but current code records materialize on
|
|
|
|
|
the current stream and only the all-reduce on the prefetch stream.
|
|
|
|
|
3. MLA and index `wait_attention_window()` tests expect the pending event to
|
|
|
|
|
be waited/launched. Current code returns without waiting in those unit
|
|
|
|
|
setups.
|
|
|
|
|
4. `test_mla_prefetch_min_prefix_pages_uses_cached_token_default_and_can_override`
|
|
|
|
|
expects `SGLANG_CP_SHARED_KV_MLA_PREFETCH_MIN_PREFIX_PAGES=-2` to return
|
|
|
|
|
`16` for `cp_size=4,page_size=64`, but current code returns the dynamic
|
|
|
|
|
token default of `8`.
|
|
|
|
|
|
|
|
|
|
Correction:
|
|
|
|
|
|
|
|
|
|
- Treat these as separate from the page-boundary extent work.
|
|
|
|
|
- The fallback-log test should be updated to the new warning prefix because C11
|
|
|
|
|
defines `[CP_SHARED_KV_FALLBACK]` warning-level logging as the intended
|
|
|
|
|
production contract.
|
|
|
|
|
- Code inspection plus the Phase 8 prefetch doc confirm the current stream/wait
|
|
|
|
|
behavior is intentional: prefix materialize runs on the current stream because
|
|
|
|
|
MLA/index materialize kernels consume SM and should remain ordered; only the
|
|
|
|
|
async all-reduce is enqueued on the prefetch stream. `wait_attention_window()`
|
|
|
|
|
is a historical hook and must not wait; the real wait remains at next-layer
|
|
|
|
|
`consume(...)` immediately before use. The stale unit tests should assert this
|
|
|
|
|
deferred-consume model instead of forcing attention-tail waits.
|
|
|
|
|
- `SGLANG_CP_SHARED_KV_MLA_PREFETCH_MIN_PREFIX_PAGES=-2` has no separate runtime
|
|
|
|
|
contract in docs or code; all negative values use the same dynamic default:
|
|
|
|
|
`max(cp_size, ceil(_MLA_PREFETCH_DEFAULT_MIN_PREFIX_TOKENS / page_size))`. The
|
|
|
|
|
expected value changed from 16 to 8 after the default token threshold was reset
|
|
|
|
|
from 1024 to 512; the stale test expectation should be updated, not the code.
|
|
|
|
|
|
|
|
|
|
Tests:
|
|
|
|
|
|
|
|
|
|
- Re-run the exact failed tests after each fix, then the three-file remote
|
|
|
|
|
suite.
|
|
|
|
|
|
|
|
|
|
## Testing requirements
|
|
|
|
|
|
|
|
|
|
Add targeted tests before implementation:
|
|
|
|
|
|
|
|
|
|
1. `extend_len % page_size != 0` MLA partial current reuse:
|
|
|
|
|
- prefix page-aligned,
|
|
|
|
|
- suffix has page slack,
|
|
|
|
|
- dense buffer includes padded suffix rows,
|
|
|
|
|
- slack locs remap to `-1`.
|
|
|
|
|
|
|
|
|
|
2. Equivalent NSA index partial reuse case:
|
|
|
|
|
- tail page included in page table,
|
|
|
|
|
- valid current tokens remap,
|
|
|
|
|
- slack positions invalid.
|
|
|
|
|
|
|
|
|
|
3. HiCache metadata:
|
|
|
|
|
- valid length and padded length differ,
|
|
|
|
|
- `page_owners` length is padded-page count,
|
|
|
|
|
- host hit exposed to scheduler is valid length.
|
|
|
|
|
|
|
|
|
|
4. Capacity:
|
|
|
|
|
- per-owner required capacity is derived from padded pages;
|
|
|
|
|
- target/draft required counts match.
|
|
|
|
|
|
|
|
|
|
5. Transfer:
|
|
|
|
|
- page list includes padded tail page;
|
|
|
|
|
- decode-visible sequence length remains valid length.
|
|
|
|
|
|
|
|
|
|
## Implementation guidance
|
|
|
|
|
|
|
|
|
|
1. Introduce a small helper/data object for page-aligned cache extents. Do not
|
|
|
|
|
pass raw `len`, `num_tokens`, or `required_slots` through new code without
|
|
|
|
|
naming whether it is valid or padded.
|
|
|
|
|
2. Update MLA prefetch compose first, because it is the observed failing path.
|
|
|
|
|
3. Update index partial reuse only after MLA has tests for slack masking.
|
|
|
|
|
4. Update HiCache metadata/capacity before enabling this contract for host-hit
|
|
|
|
|
load/backup paths.
|
|
|
|
|
5. Any temporary fallback must be a `WARNING` with a narrow reason and should be
|
|
|
|
|
removed after page-padded compose is complete.
|
|
|
|
|
|
|
|
|
|
## Non-goals for this pass
|
|
|
|
|
|
|
|
|
|
- Do not change request/logical token length semantics.
|
|
|
|
|
- Do not make `split_list` include fake tokens.
|
|
|
|
|
- Do not add a hot-path CP collective to reconcile capacity.
|
|
|
|
|
- Do not make draft KV manage an independent padded/valid contract.
|
|
|
|
|
- Do not silently fall back from the page-aligned contract.
|