Overlap CP shared KV prefix materialization for cached MLA prefill

Shared CP KV materialization remained on the critical path for cached
NSA/MLA prefill batches.  This change introduces a one-layer-ahead
prefetcher that materializes the cached prefix for the next layer on a
separate CUDA stream and consumes it when that layer reaches attention.
The prefetch path keeps the existing dense page-table semantics, defers
waiting until the prefetched buffer is actually consumed, and uses the
TAI optimized materialize/remap helpers when enabled before falling back
to the torch implementation.

The implementation is intentionally gated by environment variables and
keeps layer-2-only probe logging for functional confirmation without
making normal profiling noisy.

Constraint: Prefill CP shared KV must preserve existing page-table and dense KV semantics for NSA paged topk attention
Constraint: The production performance path requires SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE=1 and logging disabled
Rejected: Wait immediately after the producer layer attention | this truncated the overlap window and hid less work
Rejected: Torch-only prefetch materialize | it bypassed the optimized TAI materialize/remap path and could erase the expected win
Confidence: medium
Scope-risk: moderate
Directive: Do not evaluate Phase8 throughput with SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH=1; use it only to confirm create/start/consume_hit behavior
Tested: Local AST parse for modified Python files
Tested: Local git diff --check
Tested: Remote g0034 container AST parse for modified files under /sgl-workspace/sglang-tai
Tested: Remote g0034 container pytest target covering Phase8 log env, TAI range materialize, optimized slot inverse/remap, and existing token TAI path
Not-tested: Full prefill/decode/router throughput after the TAI prefetch-path fix

This commit is contained in:

laoyao0822

2026-05-03 03:09:59 +08:00

parent 5769b63082

commit bc23a81884

6 changed files with 1178 additions and 99 deletions

									
										python/sglang/srt/environ.py
									
		+3
		
												View File
												
				@@ -205,6 +205,9 @@ class Envs:

				    SGLANG_DEBUG_CP_SHARED_KV = EnvBool(False)

				    SGLANG_CP_SHARED_KV_CURRENT_REUSE = EnvBool(False)

				    SGLANG_CP_SHARED_KV_USE_TAI_MATERIALIZE = EnvBool(False)

				    SGLANG_CP_SHARED_KV_ENABLE_MLA_PREFETCH = EnvBool(False)

				    SGLANG_CP_SHARED_KV_LOG_MLA_PREFETCH = EnvBool(False)

				    SGLANG_CP_SHARED_KV_MLA_PREFETCH_WAIT_AFTER_ATTENTION = EnvBool(False)

				    SGLANG_TEST_REQUEST_TIME_STATS = EnvBool(False)

				    SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK = EnvBool(False)

				    SGLANG_SIMULATE_ACC_LEN = EnvFloat(-1)