Skip get_load() snapshot when no DP load-balancer consumes it
stream_output_generation computed a full load snapshot on every output flush: get_load() walks the waiting/bootstrap/prealloc queues summing seqlens and builds a DpRequestInfoqOutput per queued request. The only consumer is the DP load-balancer path in tokenizer_manager, which forwards it iff dp_size > 1. Gate the computation on the same condition; at dp_size == 1 the snapshot was computed and shipped for nothing on every flush. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1066,7 +1066,11 @@ class SchedulerOutputProcessorMixin:
|
||||
spec_acceptance_histogram = []
|
||||
retraction_counts = []
|
||||
output_hidden_states = None
|
||||
load = self.get_load()
|
||||
# The load snapshot is only consumed by the DP load-balancer
|
||||
# (tokenizer_manager forwards it iff dp_size > 1). get_load() walks
|
||||
# every queue and builds per-request info objects, so skip it entirely
|
||||
# when nothing will read it.
|
||||
load = self.get_load() if self.server_args.dp_size > 1 else None
|
||||
routed_experts = None
|
||||
customized_info = {}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user