leavelet
fdc23f7d61
Replaces the O(victims*candidates) per-iteration greedy argmin rescan in _plan_cp_load_back_owner_lane_evictions with a leaf-up min-heap. The score (-contribution, -unlock, slru_priority, node.id) depends on the current deficits, so a static heap is not equivalent; instead use LAZY RE-EVALUATION: a popped entry is consumed only if its score still matches the node's current score (deficits unchanged since push), else it is re-pushed with the fresh score. Stale scores are always optimistic (contribution = sum(min(counts[o], deficits[o])) and the ancestor-unlock contribution only shrink as deficits shrink), so the first up-to-date popped entry is exactly the global argmin the full rescan would have picked -> PROVABLY EQUIVALENT, with the same leaf-up eligibility + ancestor-unlock + parent-push. Determinism/rank-uniformity preserved (selection decided by the total-ordered score; the heap insertion seq only orders structurally-equal tuples). Equivalence proven by test: a verbatim reference greedy + a randomized property test (300 seeds, single- AND multi-owner deficits, non-uniform counts exercising the lazy-re-eval boundary, varied SLRU priorities) asserting byte-identical (victims, planned_freed, remaining), plus an explicit leaf-up + ancestor-unlock case (child-then-parent). 14 planner tests pass (1 pre-existing unrelated EAGLE-tail failure unchanged). Micro-bench (V100, candidates=2000 deficit=7877, benchmark/hicache/bench_cp_owner_lane_planner.py): A. ORIGINAL (.item() x cp, no memo) 50.06s B. memo + bincount (greedy) 0.494s (101x) C. lazy-re-eval heap 0.162s (309x; 3.0x over B) selection A==B==C identical Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run synthetic multi-turn benchmark
# SGLang server with radix cache disabled
python -m sglang.launch_server --model-path Qwen/Qwen2.5-14B-Instruct --port 30000 --disable-radix-cache
# SGLang server with radix cache on and first-come-first-serve policy
python -m sglang.launch_server --model-path Qwen/Qwen2.5-14B-Instruct --port 30000 --schedule-policy fcfs
# The default SGLang server with radix cache on and long-prefix-match policy
python -m sglang.launch_server --model-path Qwen/Qwen2.5-14B-Instruct --port 30000
# SGLang server with hierarchical radix cache enabled
python -m sglang.launch_server --model-path Qwen/Qwen2.5-14B-Instruct --port 30000 --enable-hierarchical-cache
python bench_multiturn.py --model-path Qwen/Qwen2.5-14B-Instruct
Note: The performance gain of hierarchical caching depends on the ratio of reusable tokens to GPU memory capacity. The more tokens to be reused, the larger the model, and the more constrained the GPU memory size, the greater the benefit one can expect from hierarchical caching.
Benchmark with more datasets
Download Dataset
./download.sh {sharegpt|ultragpt|loogle|nextqa|all}
This script will automatically download the required dataset to the current working directory
Multiturn Benchmark
Supported Datasets
- sharegpt
- ultrachat
- loogle
Example Usage:
python3 bench_serving.py --model mistralai/Mistral-7B-Instruct-v0.3 --backend sglang \
--dataset-path longdep_qa.json --dataset-name loogle --request-rate 10 --num-prompts 10 \
--port 8001 --enable-multiturn --disable-shuffle
This uses mistralai/Mistral-7B-Instruct-v0.3 model with sglang as backend. The dataset
is longdep_qa.json. We send 10 conversations with 10 req/s to port 8001. We enable
multiturn chat without shuffling the order of conversations (i.e. following the original
order in the dataset file).
Note:
The requests of multiple conversations are sent in a round robin fashion.
For example, if we have 3 conversations A, B, C whose rounds are [2, 3, 4] correspondingly,
multiturn chat will send the requests to the backend in the following order: [A1, B1, C1, A2, B2, C2, B3, C3, C4]
This has implications on the cache reuse patterns: the cache reuse distance is the largest
under this request pattern (which means a prefix-aware local scheduler in the backend can
yield the most benefit compared to a FIFO scheduler)
Shared Prefix Benchmark
Supported Datasets
- loogle
Example Usage:
python3 bench_serving.py --model mistralai/Mistral-7B-Instruct-v0.3 --backend sglang \
--dataset-path longdep_qa.json --dataset-name loogle --request-rate 10 --num-prompts 10 \
--port 8001 --enable-shared-prefix --disable-shuffle
Note:
Shared Prefix benchmark sends the questions for the same prompt together. For example,
if we have 3 shared prefix A, B, C, which have [2, 3, 4] questions correspondingly,
the shared prefix benchmark will send the requests to the
backend in the following order: [A+Q1, A+Q2, B+Q1, B+Q2, B+Q3, C+Q1, C+Q2, C+Q3].
Multi Modality Benchmark (WIP)
Supported Datasets:
- nextqa
Example Usage:
Server:
python3 -m sglang.launch_server --model-path lmms-lab/LLaVA-NeXT-Video-7B --tp 2 --dp 1 --port 8001 \
--host 0.0.0.0 --mem-fraction-static 0.9 --tokenizer-path llava-hf/llava-1.5-7b-hf \
--json-model-override-args "{\"architectures\": [\"LlavaVidForCausalLM\"], \"model_type\":\"llava\", \"mm_spatial_pool_stride\":2}"
Client:
python3 bench_serving.py --model lmms-lab/LLaVA-NeXT-Video-7B --backend sglang --dataset-path \
NExTVideo --dataset-name nextqa --request-rate 10 --num-prompts 1 --disable-shuffle --port 8001 \ --enable-multiturn --max-frames 16 --tokenizer llava-hf/llava-1.5-7b-hf --fixed-output-len 2048
Note: for the server args, tokenizer-path, overriding architecture are necessary.
Supported Backend
- sglang (oai)
- vllm (oai)
- lmdeploy (oai)