chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
Files
47cb9b9d116d1c5c112e894110ec94de80f2db6b
sglang/docs/rfc/RFC-0001.md
wxiwnd 520770da4c feat(mem-cache): enable CP shared-KV batch owner allocation 
Route supported multi-request CP shared-KV extend allocation through request-local owner plans so downstream direct-write paths can trust flattened out_cache_loc owner lanes. Unsupported page plans now fail fast instead of falling back to legacy allocation.

Include the normative govctl RFC and completed work-item trail in the tracked repository; keep transient superpowers planning outside the sglang branch.

Constraint: RFC-0001 W2 requires no legacy multi_batch fallback for supported bs>1 owner-lane allocation
Rejected: concatenating requests before owner planning | CP ownership is request-relative
Rejected: committing only docs/superpowers plan | it omits the normative RFC and work-item trail
Confidence: high
Scope-risk: moderate
Directive: Preserve flattened request order when changing CP shared-KV allocation; run govctl from the sglang repository root
Tested: remote uv test_nsa_cp_utils.py 39 OK; remote uv test_cp_shared_kv_layout.py 34 OK; remote uv test_alloc_pages_with_owners.py 10 OK; govctl check; git diff --check
Not-tested: CUDA ETE/perf paths beyond focused W1/W2 unit suites
2026-06-03 03:26:08 +08:00

145 lines
8.3 KiB
Markdown
Raw Blame History

<!-- GENERATED: do not edit. Source: RFC-0001 -->
<!-- SIGNATURE: sha256:41623be2ca618d29a789d7b0291260685b076cac42014eb3577e6df62e9f1bae -->
# RFC-0001: CP Shared-KV Batch Owner-Lane Allocation
> **Version:** 0.1.0 | **Status:** normative | **Phase:** stable
---
## 1. Summary
### [RFC-0001:C-SUMMARY] Summary (Informative) <a id="rfc-0001c-summary"></a>
This RFC specifies the W2 allocation contract for NSA prefill CP shared-KV when a real extend batch contains more than one request.
**Scope:** This RFC covers owner-lane page-owner planning, allocation ordering, owner-lane invariants, capacity failure semantics, and compatibility requirements for CP shared-KV extend allocation. It does not specify CP metadata construction, KV direct-write kernels, current reuse, HiCache load/backup, or EAGLE draft behavior except where those later workstreams depend on the allocation contract.
**Rationale:** W2 must make batch-size greater than one enter the same owner-lane allocation contract as batch-size one. Without a normative allocation contract, later direct-write and HiCache workstreams cannot trust that newly allocated logical pages belong to the ranks that compute their contents.
*Since: v0.1.0*
---
## 2. Specification
### [RFC-0001:C-REQUEST-LOCAL-OWNER-PLANNING] Request-local owner planning (Normative) <a id="rfc-0001c-request-local-owner-planning"></a>
For CP shared-KV extend allocation, the system MUST derive page-owner requirements independently for each request in the batch.
The system MUST use each request's own prefix length, extend length, page size, CP size, and in-sequence CP split semantics when deriving page owners.
The system MUST NOT derive owner requirements by treating multiple requests as one concatenated sequence.
The system MUST reject or fail fast for a request whose owner requirements cannot be derived under the CP shared-KV page-aligned contract.
**Rationale:** CP in-sequence ownership is request-relative. Combining requests before deriving owners can assign pages to the wrong compute rank even if the final flattened token count is correct.
*Since: v0.1.0*
### [RFC-0001:C-FLATTENED-OWNER-ORDER] Flattened owner order (Normative) <a id="rfc-0001c-flattened-owner-order"></a>
The system MUST concatenate per-request page-owner lists in request order.
The system MUST preserve the order of pages within each request's owner list.
The system MUST NOT reorder the flattened owner list by owner lane, physical page availability, or CP rank.
The system MUST keep the flattened owner list length equal to the number of newly allocated logical pages for the extend batch.
**Rationale:** The extend output cache location tensor is consumed in flattened request-token order. Reordering owners by lane would make page-level ownership appear locally valid while corrupting request-to-token placement.
*Since: v0.1.0*
### [RFC-0001:C-OWNER-LANE-INVARIANT] Owner-lane allocation invariant (Normative) <a id="rfc-0001c-owner-lane-invariant"></a>
For every newly allocated non-dummy logical page, the system MUST choose a logical page id whose owner lane equals the corresponding flattened expected owner.
The owner lane of a non-dummy logical page MUST be computed as `(logical_page_id - 1) % cp_size`.
The system MUST NOT allocate dummy page zero for request data.
The produced extend cache locations MUST remain in flattened request-token order.
The produced extend cache locations MUST map each newly allocated page to the expected owner lane before later CP direct-write consumers observe the locations.
**Rationale:** W3 direct write can only write local KV/index pages safely if the logical page selected during allocation is owned by the rank that computes that page.
*Since: v0.1.0*
### [RFC-0001:C-NO-LEGACY-FALLBACK] No legacy allocation fallback for supported batches (Normative) <a id="rfc-0001c-no-legacy-fallback"></a>
When CP shared-KV is enabled, context-parallel prefill is enabled, in-sequence CP split mode is selected, and all requests in the extend batch satisfy the page-aligned owner-planning contract, the system MUST use owner-lane allocation for batch-size greater than one.
The system MUST NOT use legacy paged allocation for a supported CP shared-KV batch-size greater than one case.
The system MUST NOT report `multi_batch` as a fallback reason for a supported CP shared-KV batch-size greater than one case.
If a CP shared-KV batch-size greater than one case is unsupported, the system MUST emit an explicit fail-fast or fallback reason using the CP shared-KV diagnostic naming scheme before leaving the owner-lane allocation path.
**Rationale:** Silent legacy allocation breaks the page-owner contract while allowing execution to continue into paths that assume owner-lane placement.
*Since: v0.1.0*
### [RFC-0001:C-CAPACITY-AND-EVICTION] Owner-lane capacity and eviction (Normative) <a id="rfc-0001c-capacity-and-eviction"></a>
If owner-lane allocation cannot be satisfied from immediately available owner-lane pages, the system SHOULD use the existing owner-lane eviction and retry mechanism before reporting capacity failure.
After owner-lane eviction and retry, if the required owner lanes remain unavailable, the system MUST raise a recoverable capacity-wait error rather than falling back to legacy allocation.
The capacity-wait error MUST include required, available, and deficit page counts by owner lane when those diagnostics are available.
The system MUST preserve allocator state when an owner-lane allocation attempt fails without selecting pages.
**Rationale:** Owner-lane pressure is lane-specific, not aggregate-capacity-specific. Reporting lane diagnostics preserves scheduler recoverability and prevents legacy allocation from hiding placement bugs.
*Since: v0.1.0*
### [RFC-0001:C-BS1-COMPATIBILITY] Batch-size one compatibility (Normative) <a id="rfc-0001c-bs1-compatibility"></a>
Batch-size one CP shared-KV allocation MUST continue to satisfy the existing owner-lane allocation contract.
A batch-size one request processed through the batch-aware allocation path MUST produce the same owner sequence as the existing single-request in-sequence owner planner for the same prefix length, extend length, page size, and CP size.
The system SHOULD share owner-lane allocation logic between batch-size one and batch-size greater than one to avoid long-lived divergent behavior.
**Rationale:** W2 is an extension of the current owner-lane allocation path, not a replacement with different semantics for existing single-request prefill.
*Since: v0.1.0*
### [RFC-0001:C-VERIFICATION] Verification requirements (Normative) <a id="rfc-0001c-verification"></a>
Tests MUST cover at least two requests with different prefix lengths and extend lengths.
Tests MUST verify that each newly allocated logical page in the output cache locations has the expected owner lane from the flattened owner list.
Tests MUST verify that owner-lane exhaustion for a batch-size greater than one case raises a recoverable capacity-wait error or returns an allocation failure to the caller that triggers that error path.
Tests MUST verify that batch-size one allocation remains compatible with the existing owner-lane behavior.
Tests SHOULD verify that supported batch-size greater than one CP shared-KV allocation does not call legacy allocation and does not emit a `multi_batch` fallback.
**Rationale:** These checks directly exercise the failure modes that W2 is intended to remove: cross-request owner mixing, wrong logical page owners, capacity fallback, and single-request regression.
*Since: v0.1.0*
### [RFC-0001:C-DOWNSTREAM-CONTRACT] Downstream contract (Normative) <a id="rfc-0001c-downstream-contract"></a>
After successful W2 allocation, downstream CP shared-KV direct-write consumers MAY assume that newly allocated logical pages satisfy the owner-lane invariant for their corresponding request-local page positions.
W2 MUST NOT introduce a new planner collective.
W2 MUST NOT require W3, W4, W6, or W7 consumers to recompute request-local page owners from raw prefix and extend lengths when the allocation result and batch plan metadata are already available.
**Rationale:** W2 exists to establish a stable allocation boundary. Later workstreams should validate and consume the contract instead of re-deriving allocation ownership independently.
*Since: v0.1.0*
---
## Changelog
### v0.1.0 (2026-06-03)
Initial draft
Reference in New Issue View Git Blame Copy Permalink