4f65d7a176
High cache-hit CP shared-KV batches can have small extend tokens while still carrying substantial prefix/load-back work. Add a CP-specific cached-token admission limit so operators can bound that pressure independently from extend-token batching. Constraint: The limit must not deadlock a single high-cache-hit request; it only stops adding additional requests to a non-empty batch. Constraint: Cached tokens are counted after L2 load-back planning via prefix_len, so L1 hits and successful L2 hits share one scheduler budget. Rejected: Reuse max_prefill_tokens | it limits generic input budget and does not represent cached-token work. Rejected: Count only L1 prefix before load-back | would miss L2 hit pressure, which is one of the target cases. Confidence: high Scope-risk: moderate Directive: Keep cached-token and extend-token limits separate; they bound different scheduler costs. Tested: Remote pytest targeted cached-token PrefillAdder cases: 2 passed. Tested: Remote pytest test/registered/unit/managers/test_prefill_adder.py: 18 passed. Tested: Remote ServerArgs CP validation smoke: SERVER_ARGS_CACHED_LIMIT_OK. Not-tested: Full ETE replay with a production cached-token limit value.
Registered Tests
Tests under this directory are auto-discovered by run_suite.py via CI registration decorators.
Where Should I Put My New Test?
No server / engine launch required
| What you're testing | Directory | Requires |
|---|---|---|
| Component logic in isolation (cache, scheduler, config, parser, etc.) | unit/<module>/ |
CPU or GPU |
| CUDA kernel correctness | kernels/ |
GPU |
Server / engine launch required (E2E)
| What you're testing | Directory | Requires |
|---|---|---|
| Model inference correctness | models/, 4-gpu-models/, 8-gpu-models/ |
GPU |
| Feature-specific (OpenAI API, LoRA, speculative, distributed, VLM, etc.) | openai_server/, lora/, spec/, distributed/, ... |
GPU |
| Benchmarks (performance, accuracy, stress) | benchmark/ |
GPU |
| Platform-specific | amd/, ascend/ |
Vendor GPU |
See unit/README.md for unit test conventions.