Avoid CP HiCache owner-lane planning stalls during load-back
CP HiCache load-back eviction planning previously recomputed per-owner page counts from node token tensors while scanning evictable leaves. Under shared-KV pressure this can put scheduler-side planning onto an expensive tensor-padding path and stall before load_back can complete. This stores per-CP-size owner page counts on CP HiCache metadata and uses that CPU metadata for backed/resident CP nodes. Streaming abort handling also accepts int-like status codes so abort responses do not crash on .name/.value access. Temporary debug runbooks remain ignored. Unnecessary prefill hot-path timing logs were removed before commit; owner-lane eviction now keeps warning-level output for slow planning, insufficient eviction, or remaining deficits only. Constraint: CP shared-KV cache residency is page-owner based and already records page owners in CpHiCacheNodeMetadata. Rejected: Keep verbose prefill/owner-lane timing logs | they proved the issue but add hot-path noise after validation. Confidence: medium Scope-risk: moderate Directive: Do not reintroduce tensor-derived owner counting on CP HiCache backed nodes without measuring scheduler CPU/GPU sync cost. Tested: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py python/sglang/srt/entrypoints/openai/serving_base.py python/sglang/srt/entrypoints/openai/serving_chat.py python/sglang/srt/entrypoints/openai/serving_completions.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py Tested: git diff --check Not-tested: Local pytest collection is blocked by missing starlette dependency. Not-tested: Full ETE after log cleanup; previous pre-cleanup ETE replay reached 136098.82 prompt tok/s without killing prefill.
This commit is contained in:
1
.gitignore
vendored
1
.gitignore
vendored
@@ -283,3 +283,4 @@ tai-kernel/
|
||||
|
||||
# Local CP shared-KV/GSM8K debug ledger
|
||||
docs/advanced_features/nsa_prefill_cp_gsm8k_cachehit_temp_findings.md
|
||||
docs/advanced_features/*_temp*.md
|
||||
|
||||
Reference in New Issue
Block a user