leaveletandClaude Fable 5
8979c81a22
RadixKey.__getitem__ copied the token list on every slice, and the match/insert tree walks re-slice the remaining key at every node hop, making a single match O(len * hops) — quadratic for long cached prefixes. Slices now return O(1) offset-based views over a shared backing list; the key-match and child-key functions index the backing list directly so views are never materialized on the hot path. Keys stored in tree nodes are compacted at every store site (same cost as the old copying slices), so lock-ref walks, eviction, splits, and controller-thread reads never observe a key pinning a transient backing list. Node-key compactness is enforced by a tree-walk test. Microbenchmark (64K-token full hit, page_size=64): 16-node path: 1.86 -> 0.87 ms/match (2.1x) 128-node path: 9.80 -> 1.19 ms/match (8.2x), insert re-walk 6.9x Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unit Tests
Component-level tests that do not launch a server or load model weights. Tests can use CPU or GPU — the key criterion is no server process.
Quick Start
- Find the source file under
python/sglang/srt/. - Create the corresponding test here, mirroring the source tree:
srt/mem_cache/radix_cache.py → unit/mem_cache/test_radix_cache.py srt/sampling/sampling_params.py → unit/sampling/test_sampling_params.py - Register for CI at the top of the file (after imports, before test classes):
from sglang.test.ci.ci_register import register_cpu_ci register_cpu_ci(est_time=5, suite="stage-a-test-cpu") # or: register_cuda_ci(est_time=10, suite="stage-b-test-1-gpu-small") - Run locally:
pytest test/registered/unit/ -v # all unit tests pytest test/registered/unit/mem_cache/ -v # one module - Run with coverage:
# summary pytest test/registered/unit/ --cov --cov-config=.coveragerc -v # PR incremental check (require ≥60% on changed lines) pytest test/registered/unit/ --cov --cov-config=.coveragerc --cov-report=xml diff-cover coverage.xml --compare-branch=origin/main --fail-under=60
Example
"""Unit tests for <module> — no server, no model loading."""
from sglang.test.ci.ci_register import register_cpu_ci
register_cpu_ci(est_time=5, suite="stage-a-test-cpu")
import unittest
from sglang.srt.<module> import TargetClass
from sglang.test.test_utils import CustomTestCase
class TestTargetClass(CustomTestCase):
def test_basic_behavior(self):
obj = TargetClass(...)
self.assertEqual(obj.method(), expected)
if __name__ == "__main__":
unittest.main()
Rules
- No
popen_launch_server()orEngine(...). - No model weight loading.
- Use
CustomTestCase(fromsglang.test.test_utils, adds CI retry). - Use
unittest.mockfor dependencies that are expensive to construct.