chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

make radix cache deterministic (#10721)

Browse Source 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
This commit is contained in:
Alex Chi Z
2025-10-14 15:01:52 +02:00
committed by GitHub
parent 817e46f412
commit dc965db0e0
5 changed files with 81 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
python/sglang/srt/server_args.py
View File

@@ -1381,13 +1381,6 @@ class ServerArgs:
f"Currently only {DETERMINISTIC_ATTENTION_BACKEND_CHOICES} attention backends are supported for deterministic inference."
)
# Currently, only FA3 supports radix cache. Support for other backends is in progress
if self.attention_backend != "fa3":
self.disable_radix_cache = True
logger.warning(
f"Currently radix cache is not compatible with {self.attention_backend} attention backend for deterministic inference. It will be supported in the future."
)
# Check TP size
if self.tp_size > 1:
os.environ["NCCL_ALGO"] = "allreduce:tree"