chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

Keep chunked CP prefills solo during bs>1 admission

Browse Source 
Revert the tail-chunk co-batching gate from 54c056af because allowing a continued chunk to share the next CP bs>1 batch reopens the mixed chunk/page-tail scheduler risks we are currently avoiding. Keep the independent real-prefix budget accounting so chunked requests still contribute their carried prefix to CP cached-token and buffer estimates.\n\nConstraint: Chunked-prefill requests must remain solo until the CP split/page-tail contract is revalidated for mixed batches.\nRejected: Full revert of 54c056af | it would also drop true-prefix budget accounting and under-estimate cache/buffer pressure for admitted chunks.\nConfidence: high\nScope-risk: moderate\nDirective: Do not reintroduce tail-chunk co-batching without tests covering page-tail split, CP buffer admission, and ETE chunked+cache-hit replay.\nTested: Local py_compile for environ.py, schedule_policy.py, cp_shared_kv_compose.py, test_prefill_adder.py, test_cp_shared_kv_compose_v2_8rank.py.\nTested: Remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/managers/test_prefill_adder.py -> 25 passed.\nTested: Remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 157 passed, 2 subtests passed.\nNot-tested: Full mixed replay with chunked-prefill traffic after service restart.
This commit is contained in:
laoyao0822
2026-06-13 01:28:35 +08:00
parent f290e1e35c
commit ee843a946b
5 changed files with 9 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/sglang/srt/environ.py
-6
View File
@@ -236,12 +236,6 @@ class Envs:
# Total symm slab size override in MB (both halves). 0 = pool-derived
# hard bound (logical pages x dense page unit, see design doc).
SGLANG_CP_SHARED_KV_SYMM_HEAP_MB = EnvInt(0)
# Allow a chunked prefill request (page-aligned carried prefix) to
# co-batch with other requests under CP shared-KV bs>1: a tail chunk
# leaves extend-cap headroom for short-extend cache-hit requests; a full
# chunk still ends the scan by budget. See
# docs_internal/perf/prefill-compute-intensity-plan.md S1.
SGLANG_CP_PREFILL_MIX_CHUNKED = EnvBool(False)
# Cache-affinity batch formation (plan doc S4): keep FCFS, but skip a
# COLD candidate while the batch is warm-led so cache-hit requests group
# into one dense forward instead of being split by a cold 60K chunk.