leaveletandClaude Opus 4.8
815da6c4e5
The first 3.1 (d08ee9f8d) spilled the COLDEST resident objects and held the
protect_host eviction-pin through the entire slow disk write (released only at the
durable ack). Since the coldest objects ARE eviction's victims, a B300 cachebench
wire test wedged: evict-to-fit found everything pinned (evicted=0) ->
host_reservation_failed flood -> caching broke.
Redesign (docs_internal/cp_hicache_l3_phase3_impl_design.md §R3):
- cp_l3_store: split the single spill thread into a GATHER -> WRITE pipeline. Gather
copies each owned page off the pinned slab into a slab-independent anonymous-mmap
staging buffer (fast RAM memcpy) and acks GATHER; a separate write thread does the
O_DIRECT write + fdatasync + durable LMDB put off the staged copy and acks DURABLE.
The eviction-pin releases at the GATHER ack, so the object is evictable right after
the RAM copy; the disk write completes off the staged copy even if L2 reuses the
pages. Slab-full frees its partial slot allocation (no orphan) + acks ok=False.
- hiradix: continuous PROACTIVE spill — enqueue each just-committed object to a FIFO
deque at the replicated commit frontier (_commit_pending_backup) AND at radix splits
(the freshly-committed parent half), drained in commit order (not coldest). The deque
is capped (rank-uniform) so a disk stall can't OOM it. Spilling HOT just-committed
objects (disjoint from eviction's COLD victims) means the cold tail is already
L3-durable when evicted -> eviction just drops it, never contends for the pin.
- 3-element CP-group MIN drain [gather, durable, reload]: gather-MIN -> release_host;
durable-MIN -> l3_durable, gated by an ok-AND (a second MIN over per-op ok bits) so a
write failure on any rank means NO rank marks it durable (rank-uniform durability,
placement_digest stays green).
opus-reviewed (FIX-THEN-SHIP) + independently re-verified; review fixes folded:
split-enqueue (HIGH), deque cap for the disk-stall OOM (MED), fail-soft-on-write-failure
documented (HIGH; deliberately NOT re-enqueued — re-gather would churn under slab-full,
recompute is correctness-safe), rate-limited error logs (MED/LOW). 54 L3 unit tests
green (new: 2-phase gather-before-durable ordering, slab-full fail-soft + no slot leak).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Unit Tests
Component-level tests that do not launch a server or load model weights. Tests can use CPU or GPU — the key criterion is no server process.
Quick Start
- Find the source file under
python/sglang/srt/. - Create the corresponding test here, mirroring the source tree:
srt/mem_cache/radix_cache.py → unit/mem_cache/test_radix_cache.py srt/sampling/sampling_params.py → unit/sampling/test_sampling_params.py - Register for CI at the top of the file (after imports, before test classes):
from sglang.test.ci.ci_register import register_cpu_ci register_cpu_ci(est_time=5, suite="stage-a-test-cpu") # or: register_cuda_ci(est_time=10, suite="stage-b-test-1-gpu-small") - Run locally:
pytest test/registered/unit/ -v # all unit tests pytest test/registered/unit/mem_cache/ -v # one module - Run with coverage:
# summary pytest test/registered/unit/ --cov --cov-config=.coveragerc -v # PR incremental check (require ≥60% on changed lines) pytest test/registered/unit/ --cov --cov-config=.coveragerc --cov-report=xml diff-cover coverage.xml --compare-branch=origin/main --fail-under=60
Example
"""Unit tests for <module> — no server, no model loading."""
from sglang.test.ci.ci_register import register_cpu_ci
register_cpu_ci(est_time=5, suite="stage-a-test-cpu")
import unittest
from sglang.srt.<module> import TargetClass
from sglang.test.test_utils import CustomTestCase
class TestTargetClass(CustomTestCase):
def test_basic_behavior(self):
obj = TargetClass(...)
self.assertEqual(obj.method(), expected)
if __name__ == "__main__":
unittest.main()
Rules
- No
popen_launch_server()orEngine(...). - No model weight loading.
- Use
CustomTestCase(fromsglang.test.test_utils, adds CI retry). - Use
unittest.mockfor dependencies that are expensive to construct.