laoyao0822andOmX
250fab291d
CP shared-KV bs>1 admission already bounds request count, extend tokens, cached tokens, and an estimated temporary buffer size. The estimate missed the fp32 MQA logits temporary, whose peak grows with query rows times context rows and can dominate high-cache-hit multi-request batches. Add an MQA logits peak term to the CPU-only estimator and include it in the layer-forward peak enforced by --cp-shared-kv-prefill-max-buffer-size. When SGLANG_NSA_MQA_LOGITS_CHUNK_MAX_ROWS is set, admission estimates the post-chunk peak using that row cap; otherwise it remains conservative and assumes the full extend-row count. Constraint: Scheduler admission must stay CPU-only and cannot query CUDA free memory. Rejected: Add a separate scheduler limit for MQA logits | the existing max-buffer-size knob is the right aggregate admission budget. Rejected: Use SGLANG_NSA_MQA_LOGITS_FREE_MEM_FRACTION in scheduler | that depends on runtime CUDA free memory and would make admission host-sync or stale. Confidence: medium Scope-risk: moderate Directive: Keep the estimator conservative when chunk max rows is unset; do not rely on CUDA free-memory queries in scheduler admission. Tested: Local py_compile for estimator, scheduler, schedule_policy, and estimator tests. Tested: Local pytest test_cp_shared_kv_prefill_buffer_estimator.py: 5 passed. Tested: Remote g0034 cjy-glm5-new py_compile and estimator pytest: 5 passed. Not-tested: ETE scheduler admission under high-cache-hit bs>1 traffic. Co-authored-by: OmX <omx@oh-my-codex.dev>
Registered Tests
Tests under this directory are auto-discovered by run_suite.py via CI registration decorators.
Where Should I Put My New Test?
No server / engine launch required
| What you're testing | Directory | Requires |
|---|---|---|
| Component logic in isolation (cache, scheduler, config, parser, etc.) | unit/<module>/ |
CPU or GPU |
| CUDA kernel correctness | kernels/ |
GPU |
Server / engine launch required (E2E)
| What you're testing | Directory | Requires |
|---|---|---|
| Model inference correctness | models/, 4-gpu-models/, 8-gpu-models/ |
GPU |
| Feature-specific (OpenAI API, LoRA, speculative, distributed, VLM, etc.) | openai_server/, lora/, spec/, distributed/, ... |
GPU |
| Benchmarks (performance, accuracy, stress) | benchmark/ |
GPU |
| Platform-specific | amd/, ascend/ |
Vendor GPU |
See unit/README.md for unit test conventions.