chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[fix] relax mem_fraction_static for h200 (#5893)

Browse Source 
Co-authored-by: alcanerian <alcanerian@gmail.com>
This commit is contained in:
JieXin Liang
2025-04-30 08:01:12 +08:00
committed by GitHub
parent dd408ee481
commit e4b6133b78
2 changed files with 17 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
python/sglang/srt/model_executor/cuda_graph_runner.py
View File

@@ -135,7 +135,7 @@ def get_batch_sizes_to_capture(model_runner: ModelRunner):
gpu_mem = get_device_memory_capacity()
# Batch size of each rank will not become so large when DP is on
if gpu_mem is not None and gpu_mem > 81920 and server_args.dp_size == 1:
if gpu_mem is not None and gpu_mem > 96 * 1024:
capture_bs += list(range(160, 257, 8))
if max(capture_bs) > model_runner.req_to_token_pool.size: