leavelet
6a1e862f48
专题 S4 (design: docs_internal/perf/prefill-compute-intensity-plan.md S4, amended). Under FCFS a cold request joining a warm-led batch turns a 1-2s cache-hit forward into a 5-10s one, splitting the warm work into the 新-cache-新 pattern. The policy prevents exactly that one thing: - WARM candidates always admit (into a cold-led batch they are free density — the cold extend dominates the forward anyway). - COLD admits into an empty or cold-led batch (small colds co-batch today; the FCFS head always starts a batch so the queue keeps moving). - COLD into a WARM-led batch is skipped, bounded by a per-pass window (W=16 skips), a head defer count (K=3 passes) and an age bound (T=5s). On any bound the scan STOPS instead of force-admitting: the cold waits for the same forward either way, but leads its own clean batch next pass instead of polluting this one. The skip is strictly post-match / pre-admit (after init_next_round_input, before add_one_req): no lock, no allocation, no budget mutation to unwind, and re-matching a skipped candidate next pass is exactly what the scan already does after a cap rejection. Classification is the in-scan match result (device prefix + host hit vs a 64-token floor) — under FCFS+L2 no pre-scan signal exists, so this adds zero matching work for inspected candidates. Disabled wholesale under priority scheduling (the skip must not reorder across priority classes). Three amendments vs the design draft, reasoned in the decision-table docstring: cold+cold-led admits (STOP would regress today's small-cold co-batching); starved heads STOP rather than force-admit (clean batch boundaries at identical latency); priority interaction handled by disabling rather than per-request comparison. Decision logic is a pure function with table + bounds unit tests (28/28 adder suite green). Default OFF. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unit Tests
Component-level tests that do not launch a server or load model weights. Tests can use CPU or GPU — the key criterion is no server process.
Quick Start
- Find the source file under
python/sglang/srt/. - Create the corresponding test here, mirroring the source tree:
srt/mem_cache/radix_cache.py → unit/mem_cache/test_radix_cache.py srt/sampling/sampling_params.py → unit/sampling/test_sampling_params.py - Register for CI at the top of the file (after imports, before test classes):
from sglang.test.ci.ci_register import register_cpu_ci register_cpu_ci(est_time=5, suite="stage-a-test-cpu") # or: register_cuda_ci(est_time=10, suite="stage-b-test-1-gpu-small") - Run locally:
pytest test/registered/unit/ -v # all unit tests pytest test/registered/unit/mem_cache/ -v # one module - Run with coverage:
# summary pytest test/registered/unit/ --cov --cov-config=.coveragerc -v # PR incremental check (require ≥60% on changed lines) pytest test/registered/unit/ --cov --cov-config=.coveragerc --cov-report=xml diff-cover coverage.xml --compare-branch=origin/main --fail-under=60
Example
"""Unit tests for <module> — no server, no model loading."""
from sglang.test.ci.ci_register import register_cpu_ci
register_cpu_ci(est_time=5, suite="stage-a-test-cpu")
import unittest
from sglang.srt.<module> import TargetClass
from sglang.test.test_utils import CustomTestCase
class TestTargetClass(CustomTestCase):
def test_basic_behavior(self):
obj = TargetClass(...)
self.assertEqual(obj.method(), expected)
if __name__ == "__main__":
unittest.main()
Rules
- No
popen_launch_server()orEngine(...). - No model weight loading.
- Use
CustomTestCase(fromsglang.test.test_utils, adds CI retry). - Use
unittest.mockfor dependencies that are expensive to construct.