[4/N]DP refactor: support watching mode get_load and shortest queue strategy (#10201)
This commit is contained in:
@@ -1374,3 +1374,21 @@ class BlockReqType(Enum):
|
||||
@dataclass
|
||||
class BlockReqInput:
|
||||
type: BlockReqType
|
||||
|
||||
|
||||
@dataclass
|
||||
class GetLoadReqInput:
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class GetLoadReqOutput:
|
||||
dp_rank: int
|
||||
num_reqs: int
|
||||
num_waiting_reqs: int
|
||||
num_tokens: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class WatchLoadUpdateReq:
|
||||
loads: List[GetLoadReqOutput]
|
||||
|
||||
Reference in New Issue
Block a user