chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[AMD][with CI Fix] support two batch overlapping for mori ep (#19216)

Browse Source 
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
Co-authored-by: kkHuang-amd <wunhuang@amd.com>
Co-authored-by: Feiyue Zhai <feiyue.zhai@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
This commit is contained in:
billishyahao
2026-02-25 18:14:08 +08:00
committed by GitHub
parent c4ef33862b
commit 60eeef7370
12 changed files with 732 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
python/sglang/test/bench_one_batch_server_internal.py
View File

@@ -787,17 +787,24 @@ def run_benchmark_internal(
else:
tokenizer = get_tokenizer(tokenizer_path)
# Get token capacity
internal_state = server_info.get("internal_states", [{}])
skip_token_capacity_threshold = (
internal_state[0].get("memory_usage", {}).get("token_capacity", 1000000000)
)
dp_size = internal_state[0].get("dp_size", None) or 1
# Get effective max running requests
max_running_requests_per_dp = internal_state[0].get(
"effective_max_running_requests_per_dp", -1
)
dp_size = server_info.get("dp_size", None) or 1
# Get token capacity
skip_token_capacity_threshold = 0
for i in range(dp_size):
skip_token_capacity_threshold += (
internal_state[i]
.get("memory_usage", {})
.get("token_capacity", 1000000000)
)
assert (
max_running_requests_per_dp > 0
), f"effective_max_running_requests_per_dp is not set, {max_running_requests_per_dp=}"