Commit Graph

9 Commits

Author SHA1 Message Date
3a5928ef53 feat: add page-aligned index validation for CP HiCache direct transfers
Add validate_page_aligned_token_indices utility and apply it across
HiCache write/load paths, NSA indexer transfers, and CUDA direct copy
kernels to reject malformed (partial, misaligned, non-contiguous) page
groups before they reach native transfer code. Also validate supported
CP HiCache backend/layout combinations at server startup.
2026-05-10 02:02:54 +08:00
f38937723e fix: address CP HiCache review regressions 2026-05-08 04:11:25 +08:00
da4994ec26 fix: fail fast for CP HiCache storage args 2026-05-08 00:55:35 +08:00
54f8cab7e7 test: enforce early CP HiCache storage rejection 2026-05-08 00:50:52 +08:00
ffcbf904e4 fix: reject CP shared KV with HiCache storage 2026-05-08 00:42:08 +08:00
laoyao0822
f8fca72635 Expand prefill CP KV capacity by sharding persistent NSA KV
Prefill CP previously replicated NSA/MLA persistent KV on every CP rank, so CP8 consumed eight copies of KV memory while exposing only one rank of logical cache capacity. This change splits logical KV locs from per-rank physical storage, shards MLA latent KV and NSA index K/scale by deterministic page ownership, and keeps existing NSA attention kernels working through a full-view runtime materialization layer.

Mooncake PD transfer now sends each prefill CP rank's owned physical pages with explicit logical page positions so non-CP decode can reconstruct full-layout KV. The implementation is guarded by an explicit server flag and startup checks, and the design documentation records the implemented scope, debug environment, and Phase 3 boundary.

Constraint: Phase 2 must preserve existing NSA attention/index kernels via runtime full-view materialization
Constraint: Decode side remains non-CP and receives full KV through Mooncake
Rejected: Shard-aware NSA attention in this change | belongs to Phase 3 because it requires distributed topk/softmax/output contracts
Rejected: Request-contiguous CP ownership | unstable under chunked prefill and tied to attention split mode
Confidence: medium
Scope-risk: broad
Directive: Do not enable round-robin CP shared KV without wiring runtime materialization/PD transfer contracts for that split mode
Directive: Keep SGLANG_DEBUG_CP_SHARED_KV disabled for perf measurements; it intentionally enables CUDA-syncing diagnostics
Tested: Remote py_compile for shared-KV touched Python files in g0034 container
Tested: Remote pytest selected cp_shared/shared_kv/nsa suite: 37 passed, 34 deselected
Not-tested: Full GLM5 multi-node throughput/regression run after final doc update
Not-tested: Phase 3 shard-aware runtime, round-robin CP mode, and non-Mooncake PD backends
2026-04-26 04:11:17 +08:00
Lianmin Zheng
814202704b ci: unify PR test suite naming (#21187) 2026-03-23 00:18:45 -07:00
shuwenn
6c91590e1b [HiCache] refactor: hicache normalization flow and compatibility checks (#19669) 2026-03-20 18:38:44 -07:00
Ke Bao
c3483e8e97 [CI] Move existing unit tests into unit directory (#20631) 2026-03-15 23:25:18 +08:00