# NSA Prefill CP HiCache Per-Layer Backup Design Plan

Date: 2026-05-27

Branch context: `cjy-cp-refactor` after `367dff06f3` (`Keep CP HiCache draft KV invisible until joint readiness`).

This note records the current code reading and the design constraints for turning
HiCache backup into a per-layer data-plane operation under NSA prefill CP shared
KV, HiCache, and EAGLE/MTP draft KV.  In this document, **backup** means the
D2H target/draft KV copy into host HiCache.  It is not the CLI
`--hicache-write-policy write_back` eviction policy.

## Scope

In scope:

- prefill-side CP shared KV (`--enable-nsa-prefill-cp-shared-kv`),
- HiCache L2 host backup/load/host eviction,
- target KV plus EAGLE/MTP draft KV as one logical cache object,
- write-through / write-through-selective style backup,
- per-layer D2H backup after each layer has produced immutable KV,
- low-synchronization CP rank behavior.

Out of scope for this stage:

- decode-side cache lifecycle,
- TBO/two-batch overlap scheduling,
- L3/storage write-back or prefetch,
- fully async emergency demotion for `--hicache-write-policy write_back`,
- cross-instance cache sharing.

## Current Code Facts

### Radix node state

`TreeNode` currently stores both device and host residency fields in
`python/sglang/srt/mem_cache/radix_cache.py`:

- `value`: device KV locs, full logical loc tensor in CP shared-KV mode.
- `host_value`: legacy non-CP physical host locs.
- `host_len`: logical host-backed length in CP HiCache mode.
- `cp_hicache`: CP host metadata for owned positions, page owner pattern,
  target host slots, and optional draft host slots.
- `evicted` means `value is None`.
- `backuped` only reflects `host_value is not None`, so CP code must use
  `HiRadixCache._node_backuped()` instead of the plain property.

Current CP host-valid predicate is in
`python/sglang/srt/mem_cache/hiradix_cache.py`:

```text
_node_backuped(node):
  if CP HiCache:
    host_len > 0
    required draft metadata exists
    node is not in ongoing_write_through
```

This is already close to the desired request-visible rule: in-flight backup is
not host-visible.

### Current backup path

Current CP backup path:

1. `HiRadixCache.write_backup(node)` calls `cache_controller.write(node.value)`.
2. `HiCacheController._write_cp()` treats `node.value` as full logical locs.
3. `_write_cp()` derives local owned positions through
   `CpSharedKVLayout.owned_by_this_rank()`.
4. `_write_cp()` converts local logical locs to physical locs.
5. `_write_cp()` allocates target host slots and, when draft HiCache is
   attached, draft host slots for the same local physical count.
6. Draft host allocation failure frees target host slots and returns failure.
7. Zero-owned ranks enqueue a completed no-op write ack.
8. Non-zero-owned ranks enqueue target and draft `CacheOperation`s.
9. `HiCacheController.start_writing()` merges queued ops and calls
   `backup_from_device_all_layer()` for target, then draft, on `write_stream`.
10. One `HiCacheAck` is appended for the merged logical node ids.
11. `HiRadixCache.write_backup()` stores `host_len`, `cp_hicache`, inserts the
    node into `ongoing_write_through`, and locks the device source when this is
    write-through rather than eviction-time write-back.
12. `writing_check()` polls completed acks, uses `all_reduce(MIN)` on completed
    ack prefix length, releases write locks, and makes the node host-visible by
    removing it from `ongoing_write_through`.

Important current limitation: backup is all-layer D2H.  Load is already
per-layer H2D, but backup waits until it is submitted as one all-layer transfer.

### Current load path

Current CP load path:

1. `match_prefix()` may return a host-hit node only when `_node_backuped()` is
   true.
2. `schedule_policy.add_one_req()` calls `init_load_back()` when
   `host_hit_length > 0`.
3. `load_back()` walks host-backed evicted nodes and calls
   `HiCacheController.load_cp()`.
4. `load_cp()` replays `node.cp_hicache.page_owners` with
   `alloc_pages_with_owners()` so the new logical device allocation matches the
   write-time owner pattern.
5. It queues target and draft H2D using the same owned physical device slots.
6. `start_loading()` copies per layer and completes `LayerDoneCounter` events.
7. The batch carries `hicache_consumer_index`; KV pool reads wait for the layer
   event before using that layer.

This establishes the precedent that transfer can be layer-wise while radix
state remains scheduler-owned.

### Current host eviction path

Host eviction currently only targets host-only logical leaves:

- `_update_host_leaf_status()` removes a node from `evictable_host_leaves` when
  `not node.evicted` or `node.lock_ref > 0`.
- `_evict_host_for_physical_slots()` selects from `evictable_host_leaves`, skips
  malformed/non-backed/pinned/host-referenced nodes, frees target and draft
  host slots through `evict_cp_host()`, clears `host_len`, `cp_hicache`, and
  `host_value`, then removes the logical host leaf.

Therefore, in the current code, **device-valid + host-valid nodes are normally
not host-evictable**.  Host eviction is intended for
`DEVICE_EVICTED + HOST_VALID` nodes.

### Device-hit but host-missing is valid

The design must not assume `device hit => host exists`.  The following states
are valid today and must remain valid:

- a freshly inserted node before write-through threshold is reached,
- a node whose backup failed because host was full,
- a node under in-flight backup,
- a node whose host copy was never made because the policy is selective,
- a node recomputed back into device after an earlier host-only node was evicted
  from host.

In these cases, service uses device KV directly.  Host cache is only an optional
backup copy.

## Desired State Model

Per-node host/device state should be explicit enough to separate slot ownership,
transfer progress, and request visibility:

```text
DEVICE_VALID_HOST_NONE
DEVICE_VALID_HOST_RESERVED
DEVICE_VALID_HOST_BACKUP_IN_FLIGHT
DEVICE_VALID_HOST_VALID
DEVICE_EVICTED_HOST_VALID
DEVICE_LOAD_PENDING_HOST_VALID
HOST_INVALID
```

Required visibility rules:

- `HOST_RESERVED` is not a cache hit.
- `HOST_BACKUP_IN_FLIGHT` is not a cache hit.
- `HOST_VALID` is the first state where `match_prefix()` may return a host hit.
- `HOST_RESERVED` / `HOST_BACKUP_IN_FLIGHT` must not enter
  `evictable_host_leaves`.
- Device source KV must stay protected until backup commit or rollback.
- Host destination slots must stay protected until backup commit or rollback.

The current `ongoing_write_through` map partially models this, but it mixes too
many meanings: reserved host slots, submitted D2H, pending host visibility, and
device source protection.  Per-layer backup should introduce an explicit pending
backup object or equivalent fields.

## Host-Full Reservation Flow

Host-full handling should be reserve-first, evict-on-failure, retry-once:

```text
try reserve target+draft host slots
if success:
    mark HOST_RESERVED and submit per-layer backup
else:
    evict host-only HOST_VALID victims for required local physical slots
    retry reserve target+draft host slots
    if success:
        mark HOST_RESERVED and submit per-layer backup
    else:
        skip backup; keep DEVICE_VALID_HOST_NONE
```

Rules:

1. Host eviction victims must be `DEVICE_EVICTED_HOST_VALID`, unlocked,
   unpinned, not a load source, and not a backup destination.
2. Reservation has target+draft atomic semantics.  Target allocation success and
   draft allocation failure must rollback target allocation.
3. The retry loop must be bounded.  Repeated scheduler hot retries under host
   pressure are worse than skipping backup.
4. Backup failure under write-through/selective is not a correctness failure.
   The node remains device-resident and host-missing.
5. Zero-owned ranks must still create the same logical pending backup op with a
   no-op event.

## Low-Synchronization Policy

Do not introduce per-layer collectives.

Per-layer D2H should be tracked by local CUDA events or layer completion
counters only.  CP rank synchronization should happen at coarse logical gates:

- reserve/submit failure slow path: one global success/abort check after local
  retry,
- final backup visibility commit: one final logical ack check and one batched contiguous-op prefix decision,
- debug/invariant validation: optional and off the hot path.

Fast path target:

```text
reserve local target/draft host slots
submit local per-layer D2H or zero-owned no-op
record local completion
commit completed logical prefix in deterministic order
```

Avoid:

```text
all_reduce per layer
all_reduce per node victim in a tight host-eviction loop
all_reduce every scheduler tick when no completion prefix advanced
```

Current correctness note: CP host reservation no longer uses the old
`required_host_slots` scalar `all_reduce(MAX)`.  The owner-lane vector computed
from `CpHiCacheNodeMetadata.page_owners` is now the authoritative admission
view for target and draft host capacity.  A reservation failure after this
vector predicts no deficit is treated as an invariant violation rather than
falling back to another collective.

## Per-Layer Backup Data Plane

The existing host pool exposes `load_to_device_per_layer()` but not a matching
backup method.  Per-layer backup needs a new mirror API for every relevant host
pool type:

```text
backup_from_device_per_layer(device_pool, host_indices, device_indices, layer_id, io_backend)
```

For NSA host pools this must copy both payloads for the same layer:

- MLA/NSA KV payload,
- NSA indexer K/scale payload.

The existing all-layer NSA backup calls `super().backup_from_device_all_layer()`
and `_backup_indexer_from_device_all_layer()`.  The per-layer API must mirror
that shape with `_backup_indexer_from_device_per_layer()`.

Ordering requirement:

```text
layer i KV store complete
  -> backup layer i target D2H
  -> backup layer i draft D2H when draft is attached
```

The backup hook should be a **layer-end hook**: it fires after layer `i` has
issued and ordered its device KV store.  Candidate integration points are the
attention layer forward path after `save_kv_cache` writes have been issued, or a
model runner/controller hook that receives the current layer id after the
layer's attention backend completes.  A next-layer-start hook is only an
implementation fallback because it is indirect and needs a separate final-layer
hook.  The design requirement is that backup must be ordered after the KV store
for that layer and before final host visibility.

## Radix Tree Safety

The radix tree should not be globally locked while per-layer backup runs.
Instead, pending state should be node-scoped:

- mark node as backup-pending,
- protect device source and host destination,
- block host-hit visibility,
- block host eviction of pending destination,
- block device eviction of pending source.

### Split policy

First implementation should use **defer-split-on-pending-backup** for nodes
with pending backup state.

If `_split_node()` would split a node in `HOST_RESERVED` or
`HOST_BACKUP_IN_FLIGHT`, do not split the in-flight backup op.  The request that
needs the partial-prefix split should be blocked from this scheduling attempt
and moved to a pending/waiting path, while the existing backup continues toward
its normal final ack.  After the backup commits or rolls back, the request can be
scheduled again and `_split_node()` can operate on stable node metadata.

This policy is deliberately compatible with future `bs > 1`: one request's
partial-prefix split must not mutate another request's in-flight backup state or
force all ranks to split pending target/draft host metadata.  It may delay the
new request behind D2H completion, but preserves deterministic radix structure,
owner-lane metadata, and target/draft strong sync.

Longer-term alternatives are op-aware split, pending-segment aliases, or
synchronous drain-before-split, but they are higher-risk and should not be part
of the first per-layer backup pass.

## Target/Draft Strong-Sync Contract

Draft/MTP host and device payloads are a shadow of the target radix node.
Per-layer backup must preserve this invariant:

```text
target HOST_VALID  <=> draft HOST_VALID
target DEVICE_VALID <=> draft DEVICE_VALID
```

Consequences:

- target and draft host slots are reserved together,
- target and draft layer D2H are submitted under one logical backup op,
- a target layer cannot make the node partially host-visible,
- draft failure rolls back target reservation,
- final commit requires all target and draft layer events complete,
- host eviction frees target and draft host slots together.

## Owner-Pattern Contract

CP shared KV uses logical page ownership.  Host backup metadata must preserve
write-time `page_owners`, because load-back allocates new logical device pages
and then uses `owned_positions` to select physical shards.

Per-layer backup does not change the owner pattern.  Reservation and metadata
must still be node/page based, not layer based:

```text
one CpHiCacheNodeMetadata per logical node
same owned_positions for every layer
same page_owners for load replay
same target/draft host indices for every layer payload
```

Layer granularity is a transfer scheduling property only, not a radix metadata
shape change.

## Implementation Plan

### Phase 0: Tests for current invariants

Add or extend unit tests before behavior changes:

- device-valid host-valid nodes are not host-evictable,
- device-valid host-missing nodes remain valid device hits,
- in-flight backup nodes are not host hits,
- in-flight backup nodes cannot be host-evicted,
- host reservation failure leaves node device-only,
- draft reservation failure rolls back target reservation,
- zero-owned ranks still produce a logical completion op,
- `_split_node()` on pending backup does not split the node; the triggering
  request is deferred/requeued in the chosen first-pass behavior.

### Phase 1: Explicit pending backup state

Introduce a pending backup object owned by `HiRadixCache` or `HiCacheController`:

```text
node_id
node reference
logical_len
metadata candidate
state: RESERVED | SUBMITTED | LOCAL_DONE | COMMITTED | ABORTED
target host indices
draft host indices
per-layer completion bitmap/events
```

Keep `_node_backuped()` false while the node has a pending backup object.

### Phase 2: Reservation API split

Split current `_write_cp()` into reserve and submit phases:

```text
reserve_write_cp(device_indices, node_id) -> metadata candidate or failure
submit_write_cp_all_layer(pending)        # compatibility baseline
submit_write_cp_layer(pending, layer_id)  # new per-layer path
```

The all-layer path should remain as a guarded fallback until per-layer tests are
passing, but unexpected CP+MTP invariant failures must raise rather than silently
falling back.

### Phase 3: Per-layer host-pool backup API

Add `backup_from_device_per_layer()` to host pool implementations used by this
path.  NSA implementation must include indexer K/scale per layer.

Initial tests can use fake host pools to verify call ordering and target/draft
pairing without requiring GPU kernels.

### Phase 4: Per-layer controller submission

Add a controller-side per-layer backup producer that:

- owns one transfer op per logical pending backup,
- submits target and draft copies for each layer,
- records per-layer completion events,
- exposes one final completion event / status for the logical node.

No per-layer all-reduce is allowed.

### Phase 5: Radix final commit and rollback

Move host visibility from reservation time to final commit:

- final commit writes `host_len` and `cp_hicache` or marks them visible,
- commit removes the pending op,
- commit releases the device source lock,
- rollback frees target/draft host slots and releases the device source lock,
- host eviction can only see committed `HOST_VALID` nodes.

### Phase 6: Host-full slow path

Refactor host full handling so reserve failure returns required local physical
slots, host eviction frees only valid host-only victims, and reservation retries
once.  Add logs/counters for backup skipped due to host pressure.

The current first implementation uses rank-wide
`MAX(required_host_slots)` synchronization before eviction/retry so all CP ranks
observe the same capacity-pressure branch.  This fixes correctness under
rank-skewed host pressure, but it is a known performance risk if host-full
becomes common.  Follow-up optimization should reduce how often this slow path
is reached rather than adding more collectives around it.

### Phase 7: Split safety

Implement defer/requeue-on-pending-split for nodes with pending backup.  Add
tests around partial prefix matches during in-flight backup, including a future
`bs > 1` style case where one request would split a node being backed up for
another request.

### Phase 8: Verification on remote ETE

Remote validation should run in the container, not local pytest for CUDA-heavy
paths:

- CP shared KV + HiCache + EAGLE/MTP short prompts,
- host-hit scenario with accept length stability,
- host pressure scenario that forces host eviction before reservation retry,
- zero-owned rank coverage,
- cache hit after split-like prefixes,
- no leak in target or draft host allocators after abort/rollback.

## Open Questions / Decisions

1. **Layer completion hook.**  The chosen semantic hook is explicit model
   layer-end after that layer's target/draft KV and NSA indexer writes have
   been enqueued.  Store-side notifiers are intentionally not the ownership
   boundary because MLA, NSA indexer, TAI fused store, Triton store, and
   zero-local paths can bypass one another.  The first implementation hooks
   `DeepseekV2DecoderLayer.forward()` and the TBO `op_comm_postprocess_layer()`
   path; TBO children only notify from subbatch 1 so a layer is not backed up
   after only the first child has written its KV.
2. **Pending split behavior.**  The chosen first pass is defer/requeue the
   request that would split a node with pending backup.  Do not split the
   in-flight backup op, including for future `bs > 1`.
3. **Ack batching threshold.**  `writing_check()` no longer enters the TP
   all-reduce while a write is ongoing but no final ack exists.  The remaining
   `MIN` is the final host-visibility barrier for ack entries that do exist:
   it should not run per layer, and removing it requires a different global
   visibility protocol rather than a local-only commit.
4. **Failure policy under reserve mismatch.**  Capacity pressure should skip
   backup; malformed target/draft metadata should fail fast.  If a local rank
   fails reservation after deterministic host eviction while another succeeds,
   the slow path should globally abort the backup op and rollback local slots.

## Implementation Notes: P4-P8 Pass

Implemented in the first P4-P8 pass:

- host pools now expose `backup_from_device_per_layer()`;
- controller has `submit_write_cp_layer()` with one final logical ack after all
  target/draft layers;
- `HiRadixCache.write_backup()` now reserves host slots first, keeps the node
  host-invisible in `pending_host_backups`, retries reservation once after
  host-only eviction, and commits host visibility only when the write ack
  completes;
- pending backup split attempts are deferred instead of mutating the radix tree;
- request admission marks deferred prefix matches and keeps those requests in
  the waiting path for a later retry;
- KV pools expose a no-op-safe explicit layer-end notifier.  Store methods and
  fused store kernels do not drive backup progress.

Important limitation: the current `sgl_kernel.kvcacheio` surface has H2D
per-layer page-first kernels and D2H all-layer page-first kernels, but not a
D2H per-layer LF->PF page-first kernel.  The per-layer host-pool method
therefore raises for `kernel + page_first` per-layer backup instead of silently
falling back to a slow torch copy.  The production backup path remains on
`submit_write_cp_all_layer()` as a compatibility baseline until that kernel is
added; the per-layer controller path is covered by fake/direct unit tests and is
ready for kernel integration.

## Implementation Notes: tai-kernel LF->PF pass

The missing production kernel is now supplied by `tai-kernel`:

```text
tai_kernel.nsa_prefill.transfer_kv_per_layer_mla_lf_pf(
    src_layer_buffer,
    dst_page_first_host_buffer,
    src_indices,
    dst_indices,
    layer_id,
    item_size,
    dst_layout_dim,
)
```

This covers the GLM5/NSA production shape:

- MLA/NSA target KV page-first host backup;
- NSA indexer page-first host backup;
- target and draft backup through the same logical reservation/ack path.

MHA page-first remains fail-fast until it has a dedicated kernel.

The first production forward-overlap path reserves host slots after
`prepare_for_extend()` and before the model forward, then calls
`submit_write_cp_per_layer(..., catch_up_all_layers=False)`.  Radix insertion
only attaches that prepared reservation; it does not replay an all-layer
catch-up copy.  The model layer-end hook is the forward progress driver:

```text
submit_write_cp_per_layer(..., catch_up_all_layers=False)
  -> DeepseekV2 layer-end calls token_to_kv_pool.notify_layer_end_for_backup(layer_id)
  -> HiCacheController.on_layer_end(layer_id, source="target" | "draft")
  -> submit_write_cp_layer(reservation, layer_id, submit_target=..., submit_draft=...)
  -> final ack only after all target/draft layers are submitted
```

This keeps the public state model unchanged: reserved or partially copied host
payloads remain invisible until the final write ack commits
`pending_host_backups`.  Target and draft layer readiness are tracked
separately so an early target hook cannot copy draft KV before draft has stored
the same layer.

The previous store-notifier attempt was rejected as the primary correctness
boundary.  It was too easy for fused paths to skip it: NSA fused indexer store
bypassed `set_index_k_scale_buffer()`, MLA can use `SGLANG_CP_SHARED_KV_FUSED_MLA_STORE=1`,
and zero-local paths do not necessarily touch the same setter.  Missing one
store notifier left the prepared backup in `pending_layer_writes` forever: the
first request could finish prefill without a final write ack, and a second
identical request could hit the pending node and wedge the scheduler.  The
layer-end hook makes backup progress independent of the chosen store kernel;
store-side hooks should remain absent or debug-only.

One more correctness/performance edge was observed with repeated EAGLE requests.
Scheduler prefix matching and final radix insertion can have different effective
lengths because matching is based on the schedulable prefix while insertion uses
the final page-aligned key.  With EAGLE bigram keys this can leave
`req.cache_protected_len` shorter than the prefix that insertion will find in the
tree.  If early backup reserves `[cache_protected_len, insertion_key_len)` blindly,
the system copies a duplicate page, then final insertion discovers that no new
node was created and rolls the prepared backup back with `reason=insert_miss`.
This is safe but wastes host reservation, per-layer callbacks, and write-stream
work on repeated prompts.

The early backup path now probes the existing radix prefix for the final
insertion key without splitting the tree, and only reserves the suffix beyond
`max(cache_protected_len, existing_insertion_prefix_len)`.  This preserves the
cold-request behavior while avoiding duplicate prepared backups on exact repeats.

## 0SM Backup Direction: Direct Page-First-Direct

If the goal is to make backup overlap forward compute without stealing SMs, the
preferred direction is **not** to further optimize the `kernel + page_first`
backup kernels.  Those paths are CUDA kernels by construction, including the
current `tai-kernel` per-layer LF->PF implementation, and therefore consume SM
resources.

The 0SM-oriented path should be:

```text
--hicache-io-backend direct
--hicache-mem-layout page_first_direct
```

In this mode, all-layer LF->PF backup already has a direct-copy implementation
through `sgl_kernel.kvcacheio.transfer_kv_all_layer_direct_lf_pf()`, which uses
`cudaMemcpyBatchAsync` when the CUDA runtime/driver supports it and falls back to
non-blocking copy slices otherwise.  This is the copy-engine style path and is
the right base for 0SM backup.

The current gap is per-layer D2H backup.  `sgl_kernel.kvcacheio` exposes
`transfer_kv_per_layer_direct_pf_lf()` for page-first-direct host-to-device load,
and `transfer_kv_all_layer_direct_lf_pf()` for all-layer device-to-host backup,
but it does not expose the per-layer device-to-host sibling:

```text
transfer_kv_per_layer_direct_lf_pf(
    src_ptrs,      # one target/draft device layer, or K/V layer pair
    dst_ptrs,      # page_first_direct host pool
    src_indices,   # local physical device token/page spans
    dst_indices,   # reserved host token/page spans
    layer_id,
    page_size,
)
```

That operator can reuse the existing internal page-first-direct direct-copy
implementation with `IsLf2Pf=true` and `start_layer_id=layer_id`.  It should
require page-aligned contiguous spans and fail fast if the layout/backend does
not match.  After this operator exists, the current Python-loop fallbacks in
`memory_pool_host.py` for `direct + page_first_direct` per-layer backup should
route to it for:

- MLA/NSA target KV,
- EAGLE/MTP draft KV,
- NSA indexer state.

### Backend split status

The current controller has one `io_backend` field, and uses it for both load and
backup:

```text
start_loading()  -> load_to_device_per_layer(..., self.io_backend)
start_writing()  -> backup_from_device_all_layer(..., self.io_backend)
submit_write_cp_layer() -> backup_from_device_per_layer(..., self.io_backend)
```

It also uses the same backend in `move_indices()` to decide whether indices live
on CUDA tensors (`kernel`) or CPU tensors (`direct`).  Therefore, the current
runtime supports `direct` as a whole HiCache CPU/GPU I/O mode, but does not
support a pure configuration knob for "backup direct, load kernel".

Splitting load and backup backends is possible, but should be treated as a
second step rather than the first 0SM backup step.  The reason is layout
compatibility: `direct` backup wants `page_first_direct`, while the existing
`kernel` load path supports `page_first`, not `page_first_direct`.  A split mode
would therefore also need either a `kernel + page_first_direct` load path or a
separate layout conversion path, which adds more correctness surface than first
making full direct mode work well.

Recommended sequence:

1. Implement `transfer_kv_per_layer_direct_lf_pf()` in `sgl-kernel`.
2. Route `direct + page_first_direct` per-layer target/draft/indexer backup to
   that operator.
3. Validate full direct mode for CP shared KV + HiCache + MTP/EAGLE.
4. Only if direct load is a bottleneck, introduce separate
   `backup_io_backend`/`load_io_backend` fields and the extra layout support
   required by a mixed direct-backup/kernel-load mode.

## Summary

Per-layer backup should be implemented as a data-plane refinement, not a radix
metadata shape change:

```text
radix state synchronous
host reservation explicit and pending
D2H backup per layer and local-event driven
host visibility only after final logical commit
target/draft atomic
owner pattern preserved
host-full path evicts host-only valid victims then retries reservation once
```

The main correctness boundary is request visibility.  A reserved or partially
copied host payload is not a cache hit.  The main performance boundary is
synchronization.  Per-layer backup must not add per-layer CP collectives; it
should reduce transfer tail while keeping final commit batched and deterministic.
