Commit Graph

4 Commits

Author SHA1 Message Date
laoyao0822
ceb5345410 Account decode handoff queues in DP load snapshots
Decode DP dispatch was collapsing onto a few ranks because the controller only randomizes among exact minimum load pairs. The load snapshot undercounted decode handoff work: pending prefill-info requests were absent, and DecodeRequest wrappers in prealloc/transfer queues were skipped because their rid lives on .req.

This makes scheduler load accounting unwrap DecodeRequest items and include pending decode requests, so TOTAL_TOKENS sees queued handoff backlog instead of repeatedly treating busy ranks as empty.

Constraint: Do not mask imbalance with synthetic per-request token penalties; dispatch should be driven by accurate observed load.

Rejected: Add req*4000 or other queue penalties | heuristic, workload-dependent, and hides the accounting bug.

Confidence: medium

Scope-risk: moderate

Directive: Any new decode handoff queue must be included in get_load() or DP routing can regress to stale/min-load collapse.

Tested: Remote cjy-glm5-new: PYTHONPATH=python python -m pytest -q test/registered/unit/observability/test_scheduler_metrics_load.py test/registered/unit/managers/test_prefill_adder.py -> 27 passed.

Not-tested: Fresh decode ETE distribution after service restart.
2026-06-13 02:35:10 +08:00
Lianmin Zheng
814202704b ci: unify PR test suite naming (#21187) 2026-03-23 00:18:45 -07:00
Lianmin Zheng
cea79530f8 ci: merge CPU default and stage-a-cpu-only suites (#20992) 2026-03-20 00:04:55 -07:00
Ke Bao
c3483e8e97 [CI] Move existing unit tests into unit directory (#20631) 2026-03-15 23:25:18 +08:00