chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[quantization] AWQ Marlin doesn't work when dtype is bfloat16 (#11494)

Browse Source 
Signed-off-by: Kai-Hsun Chen <khchen@x.ai>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
This commit is contained in:
Kai-Hsun Chen
2025-10-26 00:49:45 -07:00
committed by GitHub
parent 8491c794ad
commit 6371f7af27
4 changed files with 42 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
python/sglang/srt/model_executor/model_runner.py
View File

@@ -527,7 +527,7 @@ class ModelRunner:
quantization_config := getattr(
self.model_config.hf_config, "quantization_config", None
)
) is not None:
) is not None and "weight_block_size" in quantization_config:
weight_block_size_n = quantization_config["weight_block_size"][0]
if self.tp_size % self.moe_ep_size != 0: