Files
sglang/python
laoyao0822 ceb5345410 Account decode handoff queues in DP load snapshots
Decode DP dispatch was collapsing onto a few ranks because the controller only randomizes among exact minimum load pairs. The load snapshot undercounted decode handoff work: pending prefill-info requests were absent, and DecodeRequest wrappers in prealloc/transfer queues were skipped because their rid lives on .req.

This makes scheduler load accounting unwrap DecodeRequest items and include pending decode requests, so TOTAL_TOKENS sees queued handoff backlog instead of repeatedly treating busy ranks as empty.

Constraint: Do not mask imbalance with synthetic per-request token penalties; dispatch should be driven by accurate observed load.

Rejected: Add req*4000 or other queue penalties | heuristic, workload-dependent, and hides the accounting bug.

Confidence: medium

Scope-risk: moderate

Directive: Any new decode handoff queue must be included in get_load() or DP routing can regress to stale/min-load collapse.

Tested: Remote cjy-glm5-new: PYTHONPATH=python python -m pytest -q test/registered/unit/observability/test_scheduler_metrics_load.py test/registered/unit/managers/test_prefill_adder.py -> 27 passed.

Not-tested: Fresh decode ETE distribution after service restart.
2026-06-13 02:35:10 +08:00
..
2026-06-10 05:54:43 +08:00