ddc1233955
CP shared-KV bs>1 can build large fp32 MQA-logits temporaries from DeepGEMM fp8_mqa_logits. The official SGLang path already chunks normal NSA MQA logits by query rows behind a cached memory budget; carry the same budget control into our NSA indexer and extend it to CP-ragged topk paths that use row-wise topk_indices_offset_override. This keeps the previous one-time cached memory-budget behavior rather than the recent current-free-mem per-forward variant that regressed performance. A new optional max-rows env provides an explicit hard cap for debugging or controlled ETE runs without adding host syncs. Constraint: DeepGEMM materializes fp32 [q, k] logits internally, so row chunking is the narrowest way to cap temporary memory Rejected: Restore the reverted syh current-free-mem implementation | it changed hot-path heuristics and showed poor runtime performance Rejected: Split by K/context dimension | would change topk semantics and require a different transform contract Confidence: medium Scope-risk: moderate Directive: CP-ragged chunking relies on topk_indices_offset_override being row-addressed; do not route non-ragged CP paths through it without separate validation Tested: Local py_compile for environ.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py Tested: Remote g0034 cjy-glm5-new py_compile for environ.py, nsa_indexer.py, and test_cp_shared_kv_runtime.py Tested: Remote pytest TestCpSharedKVTaiMaterializeIntegration, 17 passed Not-tested: CUDA ETE high-cache-hit bs>1 workload memory/performance after chunking Co-authored-by: OmX <omx@oh-my-codex.dev>
Registered Tests
Tests under this directory are auto-discovered by run_suite.py via CI registration decorators.
Where Should I Put My New Test?
No server / engine launch required
| What you're testing | Directory | Requires |
|---|---|---|
| Component logic in isolation (cache, scheduler, config, parser, etc.) | unit/<module>/ |
CPU or GPU |
| CUDA kernel correctness | kernels/ |
GPU |
Server / engine launch required (E2E)
| What you're testing | Directory | Requires |
|---|---|---|
| Model inference correctness | models/, 4-gpu-models/, 8-gpu-models/ |
GPU |
| Feature-specific (OpenAI API, LoRA, speculative, distributed, VLM, etc.) | openai_server/, lora/, spec/, distributed/, ... |
GPU |
| Benchmarks (performance, accuracy, stress) | benchmark/ |
GPU |
| Platform-specific | amd/, ascend/ |
Vendor GPU |
See unit/README.md for unit test conventions.