[NPU] NPU quantization refactoring & more quantization formats support (#14504)
Co-authored-by: TamirBaydasov <mr.jeijy@gmail.com> Co-authored-by: Tamir Baydasov <41994229+TamirBaydasov@users.noreply.github.com> Co-authored-by: Савкин Артем <savkinartem@MacBook-Air-Viktoria.local> Co-authored-by: Edward Shogulin <edward.shogulin@gmail.com>
This commit is contained in:
co-authored by
TamirBaydasov
Tamir Baydasov
Савкин Артем
Edward Shogulin
parent
f091858304
commit
424a380077
@@ -30,7 +30,6 @@ python3 -m sglang.launch_server \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--watchdog-timeout 9000 \
|
||||
--cuda-graph-bs 8 16 24 28 32 \
|
||||
--mem-fraction-static 0.68 \
|
||||
@@ -88,7 +87,6 @@ python -m sglang.launch_server \
|
||||
--mem-fraction-static 0.6 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--max-running-requests 8 \
|
||||
--context-length 8192 \
|
||||
--disable-radix-cache \
|
||||
@@ -144,7 +142,6 @@ python -m sglang.launch_server \
|
||||
--max-running-requests 352 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep \
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
@@ -216,7 +213,6 @@ do
|
||||
--mem-fraction-static 0.81 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--max-running-requests 8 \
|
||||
--context-length 8192 \
|
||||
--disable-radix-cache \
|
||||
@@ -279,7 +275,6 @@ do
|
||||
--max-running-requests 832 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep \
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
|
||||
Reference in New Issue
Block a user