chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Update model and feature support for Ascend NPU (#16003)

Browse Source
This commit is contained in:
Hexq0210
2025-12-29 15:16:25 +08:00
committed by GitHub
parent 097330909f
commit f784cbfa92
4 changed files with 578 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
docs/platforms/ascend_npu.md
View File

@@ -3,7 +3,7 @@
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
## Installing SGLang
## Preparing the Running Environment
### Method 1: Installing from source with prerequisites
@@ -98,11 +98,17 @@ pip install -e python[srt_npu]
```
### Method 2: Using docker
__Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.
__Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.
#### Obtain Image
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
1. Download SGLang image
```angular2html
dockerhub: docker.io/lmsysorg/sglang:$tag
# Main-based tag, change main to specific version like v0.5.6,
# you can get image for specific version
Atlas 800I A3 : {main}-cann8.3.rc2-a3
Atlas 800I A2: {main}-cann8.3.rc2-910b
```
2. Build an image based on Dockerfile
```shell
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
@@ -110,6 +116,14 @@ cd sglang/docker
# Build the docker image
docker build -t <image_name> -f npu.Dockerfile .
```
#### Create Docker
__Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.
__Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.
```shell
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
@@ -122,6 +136,7 @@ alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-siz
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
# Add HF_TOKEN env for download model by SGLang
drun --env "HF_TOKEN=<secret>" \
<image_name> \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000
@@ -157,3 +172,76 @@ sudo sysctl -w vm.swappiness=10
# Check
cat /proc/sys/vm/swappiness # shows 10
```
## Running SGLang Service
### Running Service For Large Language Models
#### PD Mixed Scene
```shell
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000
```
#### PD Separation Scene
Launch prefill server
```shell
export SGLANG_SET_CPU_AFFINITY=1
# PIP: recommended to config first Prefill Server IP, all server need to be config the same ip, PORT: one free port
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode prefill \
--disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8995 \
--attention-backend ascend \
--device npu \
--base-gpu-id 0 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8000
```
Launch Decode server
```shell
export SGLANG_SET_CPU_AFFINITY=1
# PIP: recommended to config first Prefill Server IP, all server need to be config the same ip, PORT: one free port
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode decode \
--disaggregation-transfer-backend ascend \
--attention-backend ascend \
--device npu \
--base-gpu-id 1 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8001
```
Launch Router
```shell
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://127.0.0.1:8000 8995 \
--decode http://127.0.0.1:8001 \
--host 127.0.0.1 \
--port 6688
```
### Running Service For Multimodal Language Models
#### PD Mixed Scene
```shell
python3 -m sglang.launch_server \
--model-path Qwen3-VL-30B-A3B-Instruct \
--host 127.0.0.1 \
--port 8000 \
--tp 4 \
--device npu \
--attention-backend ascend \
--mm-attention-backend ascend_attn \
--disable-radix-cache \
--trust-remote-code \
--enable-multimodal \
--sampling-backend ascend
```

2
docs/platforms/ascend_npu_support.rst
View File

@@ -5,5 +5,7 @@ Ascend NPUs
:maxdepth: 1
ascend_npu.md
ascend_npu_support_models.md
ascend_npu_support_features.md
ascend_npu_deepseek_example.md
ascend_npu_qwen3_examples.md

399
docs/platforms/ascend_npu_support_features.md Normal file
View File

@@ -0,0 +1,399 @@
# Support Features on Ascend NPU
This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any
questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
## Model and tokenizer
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--model-path`<br>`--model` | The path of the model weights. This can be a local folder or a Hugging Face repo ID. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-path` | The path of the tokenizer. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-mode` | Tokenizer mode. 'auto' will use the fast tokenizer if available, and 'slow' will always use the slow tokenizer. | `auto` | `auto`, `slow` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-worker-num` | The worker num of the tokenizer manager. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--skip-tokenizer-init` | If set, skip init tokenizer and pass input_ids in generate request. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--trust-remote-code` | Whether or not to allow for custom models defined on the Hub in their own modeling files. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--context-length` | The model's maximum context length. Defaults to None (will use the value from the model's config.json instead). | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--is-embedding` | Whether to use a CausalLM as an embedding model. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-multimodal` | Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen | `None` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--revision` | The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--model-impl` | Which implementation of the model to use. * “auto” will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available. * “sglang” will use the SGLang model implementation. * “transformers” will use the Transformers model implementation.* "mindspore" will use the MindSpore model implementation. | `None` | 'auto', 'sglang', 'transformers' | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--model-loader-extra-config` | Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. | {} | Type: str' | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## HTTP server
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--host` | The host of the HTTP server. | `127.0.0.1` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--port` | The port of the HTTP server. | `30000` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--skip-server-warmup` | If set, skip warmup. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--warmups` | Specify custom warmup functions (csv) to run before server starts eg. --warmups=warmup_name1,warmup_name2 will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--nccl-port` | The port for NCCL distributed environment setup. Defaults to a random port. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Quantization and data type
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--dtype` | Data type for model weights and activations. * "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. * "half" for FP16. Recommended for AWQ quantization. * "float16" is the same as "half". * "bfloat16" for a balance between precision and range. | `auto` | `auto`, `float16`, `bfloat16` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--quantization` | The quantization method. | `None` | `modelslim` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--quantization-param-path` | Path to the JSON file containing the KV cache scaling factors. This should generally be supplied, when KV cache dtype is FP8. Otherwise, KV cache scaling factors default to 1.0, which may cause accuracy issues. | `None` | Type: Optional[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--kv-cache-dtype` | Data type for kv cache storage. "auto" will use model data type. "fp8_e5m2" and "fp8_e4m3" is supported for CUDA 11.8+. | `auto` | `auto` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Memory and scheduling
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--mem-fraction-static` | The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors. | `None` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-running-requests` | The maximum number of running requests. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--prefill-max-requests` | The maximum number of requests in a prefill batch. If not specified, there is no limit.. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-queued-requests` | The maximum number of queued requests. This option is ignored when using disaggregation-mode. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-total-tokens` | The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction. This option is typically used for development and debugging purposes. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--chunked-prefill-size` | The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-prefill-tokens` | The maximum number of tokens in a prefill batch. The real bound will be the maximum of this value and the model's maximum context length. | `16384` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--schedule-policy` | The scheduling policy of the requests. | `fcfs` | `lpm`, `fcfs` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-priority-scheduling` | Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--schedule-low-priority-values-first` | If specified with --enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--priority-scheduling-preemption-threshold` | Minimum difference in priorities for an incoming request to have to preempt running request(s). | `10` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--schedule-conservativeness` | How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently. | `1.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--page-size` | The number of tokens in a page, auto set 128 to Ascend NPU. | `128` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hybrid-kvcache-ratio` | Mix ratio in [0,1] between uniform and hybrid kv buffers (0.0 = pure uniform: swa_size / full_size = 1)(1.0 = pure hybrid: swa_size / full_size = local_attention_size / context_length) | `None` | Optional[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--swa-full-tokens-ratio` | The ratio of SWA layer KV tokens / full layer KV tokens, regardless of the number of swa:full layers. It should be between 0 and 1. E.g. 0.5 means if each swa layer has 50 tokens, then each full layer has 100 tokens. | `0.8` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-hybrid-swa-memory` | Disable the hybrid SWA memory. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Runtime options
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--device` | The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu'). Defaults to auto-detection if not specified. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tensor-parallel-size`<br>`--tp-size` | The tensor parallelism size. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--pipeline-parallel-size`<br>`--pp-size` | The pipeline parallelism size. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--pp-max-micro-batch-size` | The maximum micro batch size in pipeline parallelism. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--stream-interval` | The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--stream-output` | Whether to output as a sequence of disjoint segments. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--random-seed` | The random seed. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--constrained-json-whitespace-pattern` | (outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model to generate consecutive whitespaces, set the pattern to [\n\t ]* | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--constrained-json-disable-any-whitespace` | (xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--watchdog-timeout` | Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging. | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--soft-watchdog-timeout` | Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging. | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dist-timeout` | Set timeout for torch.distributed initialization. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--base-gpu-id` | The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine. | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--gpu-id-step` | The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,... | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--sleep-on-idle` | Reduce CPU usage when sglang is idle. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--mm-process-config` | A JSON string for multimodal preprocessing configuration. It can contain keys: `image`, `video`, `audio`. | `{}` | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Logging
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--log-level` | The logging level of all loggers. | `info` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--log-level-http` | The logging level of HTTP server. If not set, reuse --log-level by default. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--log-requests` | Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--log-requests-level` | 0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output. | `2` | `0`, `1`, `2`, `3` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--crash-dump-folder` | Folder path to dump requests from the last 5 min before a crash (if any). If not specified, crash dumping is disabled. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--crash-on-nan` | Crash the server on nan logprobs. | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-metrics` | Enable log prometheus metrics. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-metrics-for-all-schedulers` | Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp_attention is enabled, as otherwise all metrics appear to come from TP 0. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--tokenizer-metrics-custom-labels-header` | Specify the HTTP header for passing custom labels for tokenizer metrics. | `x-custom-labels` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--tokenizer-metrics-allowed-custom-labels` | The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., {'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set. | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--bucket-time-to-first-token` | The buckets of time to first token, specified as a list of floats. | `None` | List[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--bucket-inter-token-latency` | The buckets of inter-token latency, specified as a list of floats. | `None` | List[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--bucket-e2e-request-latency` | The buckets of end-to-end request latency, specified as a list of floats. | `None` | List[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--collect-tokens-histogram` | Collect prompt/generation tokens histogram. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--prompt-tokens-buckets` | The buckets rule of prompt tokens. Supports 3 rule types: 'default' uses predefined buckets; 'tse <middle> <base> <count>' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets [984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom <value1> <value2> ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--generation-tokens-buckets` | The buckets rule for generation tokens histogram. Supports 3 rule types: 'default' uses predefined buckets; 'tse <middle> <base> <count>' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets [984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom <value1> <value2> ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--gc-warning-threshold-secs` | The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable. | `0.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--decode-log-interval` | The log interval of decode batch. | `40` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-request-time-stats-logging` | Enable per request time stats logging | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--kv-events-config` | Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-trace` | Enable opentelemetry trace | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--oltp-traces-endpoint` | Config opentelemetry collector endpoint if --enable-trace is set. format: <ip>:<port> | `localhost:4317` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## API related
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|----------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--api-key` | Set API key of the server. It is also used in the OpenAI API compatible server. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--served-model-name` | Override the model name returned by the v1/models endpoint in OpenAI API server. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--weight-version` | Version identifier for the model weights. Defaults to 'default' if not specified. | `default` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--chat-template` | The buliltin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--completion-template` | The buliltin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-cache-report` | Return number of cached tokens in usage.prompt_tokens_details for each openai request. | `True` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--reasoning-parser` | Specify the parser for reasoning models. Supported parsers: [deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3]. | `None` | `deepseek-r1`, `deepseek-v3`, `glm45`, `gpt-oss`, `kimi`, `qwen3`, `qwen3-thinking`, `step3` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tool-call-parser` | Specify the parser for handling tool-call interactions. Supported parsers: [ llama3,qwen]. | `None` | `llama3`,`qwen` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--sampling-defaults` | Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top_p=1.0, etc.). 'model' uses the model's generation_config.json to get the recommended sampling parameters if available. Default is 'model'. | `model` | `openai`, `model` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--tool-server` | Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no tool server will be used. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Data parallelism
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|---------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--data-parallel-size`<br>`--dp-size` | The data parallelism size. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--load-balance-method` | The load balancing strategy for data parallelism. The Minimum Token algorithm can only be used when DP attention is applied. This algorithm performs load balancing based on the real-time token load of the DP workers. | `round_robin` | `round_robin`, `shortest_queue`, `minimum_tokens` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--prefill-round-robin-balance` | Prefill is round robin balanced. This is used to promise decode server can get the correct dp rank. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Multi-node distributed serving
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|------------------------------------------|------------------------------------------------------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
| `--dist-init-addr`<br>`--nccl-init-addr` | The host address for initializing distributed backend (e.g., `192.168.0.2:25000`). | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--nnodes` | The number of nodes. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--node-rank` | The node rank. | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Model override args
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-------------------------------|-----------------------------------------------------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
| `--json-model-override-args` | A dictionary in JSON string format used to override default model configurations. | `{}` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--preferred-sampling-params` | json-formatted sampling settings that will be returned in /get_model_info | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## LoRA
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--enable-lora` | Enable LoRA support for the model. This argument is automatically set to `True` if `--lora-paths` is provided for backward compatibility. | `False` | Bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-lora-rank` | The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--lora-target-modules` | The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. You can also set it to `all` to enable LoRA for all supported modules; note this may introduce minor performance overhead. | `None` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, `qkv_proj`, `gate_up_proj`, `all` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--lora-paths` | The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: `<PATH>` \| `<NAME>=<PATH>` \| JSON with schema `{"lora_name": str, "lora_path": str, "pinned": bool}`. | `None` | Type: List[str] / JSON objects | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-loras-per-batch` | Maximum number of adapters for a running batch, including base-only requests. | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--max-loaded-loras` | If specified, limits the maximum number of LoRA adapters loaded in CPU memory at a time. Must be ≥ `--max-loras-per-batch`. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--lora-eviction-policy` | LoRA adapter eviction policy when the GPU memory pool is full. | `lru` | `lru`, `fifo` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--lora-backend` | Choose the kernel backend for multi-LoRA serving. | `triton` | `triton`, `csgmv` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--max-lora-chunk-size` | Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when `--lora-backend` is `csgmv`. Larger values may improve performance. | `16` | `16`, `32`, `64`, `128` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Kernel Backends (Attention, Sampling, Grammar, GEMM)
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--attention-backend` | Choose the kernels for attention layers. | `None` | `ascend` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--prefill-attention-backend` | Choose the kernels for prefill attention layers (have priority over --attention-backend). | `None` | `ascend` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--decode-attention-backend` | Choose the kernels for decode attention layers (have priority over --attention-backend). | `None` | `ascend` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--sampling-backend` | Choose the kernels for sampling layers. | `None` | `pytorch`,`ascend` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--grammar-backend` | Choose the backend for grammar-guided decoding. | `None` | `xgrammar` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--mm-attention-backend` | Set multimodal attention backend. | `None` | `ascend_attn` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--nsa-prefill-backend` | Choose the NSA backend for the prefill stage (overrides `--attention-backend` when running DeepSeek NSA-style attention). | `flashmla_sparse` | `flashmla_sparse`, `flashmla_decode`, `fa3`, `tilelang`, `aiter` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--nsa-decode-backend` | Choose the NSA backend for the decode stage when running DeepSeek NSA-style attention. Overrides `--attention-backend` for decoding. | `flashmla_kv` | `flashmla_prefill`, `flashmla_kv`, `fa3`, `tilelang`, `aiter` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--fp8-gemm-backend` | Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (optimal for Blackwell and low-latency), 'cutlass' (optimal for Hopper/Blackwell GPUs and high-throughput), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). **NOTE**: This replaces the deprecated environment variables SGLANG_ENABLE_FLASHINFER_FP8_GEMM and SGLANG_SUPPORT_CUTLASS_BLOCK_FP8. | `auto` | `auto`, `deep_gemm`, `flashinfer_trtllm`, `cutlass`, `triton`, `aiter` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-flashinfer-autotune` | Flashinfer autotune is enabled by default. Set this flag to disable the autotune. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Speculative decoding
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|-----------|---------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--speculative-algorithm` | Speculative algorithm. | `None` | `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-model-path`<br>`--speculative-draft-model` | The path of the draft model weights. This can be a local folder or a Hugging Face repo ID. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-model-revision` | The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-num-steps` | The number of steps sampled from draft model in Speculative Decoding. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-eagle-topk` | The number of tokens sampled from the draft model in eagle2 each step. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-num-draft-tokens` | The number of tokens sampled from the draft model in Speculative Decoding. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-accept-threshold-single` | Accept a draft token if its probability in the target model is greater than this threshold. | `1.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-accept-threshold-acc` | The accept probability of a draft token is raised from its target probability p to min(1, p / threshold_acc). | `1.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-token-map` | The path of the draft model's small vocab table. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-attention-mode` | Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of 'prefill' (default) or 'decode'. | `prefill` | `prefill`, `decode` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-moe-runner-backend` | MOE backend for EAGLE speculative decoding, see --moe-runner-backend for options. Same as moe runner backend if unset. | `None` | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-moe-a2a-backend` | MOE A2A backend for EAGLE speculative decoding, see --moe-a2a-backend for options. Same as moe a2a backend if unset. | `None` | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Ngram speculative decoding
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------------------|-----------------------------------------------------------------------------------|------------|---------------|:--------------------------------------:|:--------------------------------------:|
| `--speculative-ngram-min-match-window-size` | The minimum window size for pattern matching in ngram speculative decoding. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-ngram-max-match-window-size` | The maximum window size for pattern matching in ngram speculative decoding. | `12` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-ngram-min-bfs-breadth` | The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-ngram-max-bfs-breadth` | The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding. | `10` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-ngram-match-type` | The match type for cache tree. | `BFS` | `BFS`, `PROB` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-ngram-branch-length` | The branch length for ngram speculative decoding. | `18` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-ngram-capacity` | The cache capacity for ngram speculative decoding. | `10000000` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Expert parallelism
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|-----------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--expert-parallel-size`<br>`--ep-size`<br>`--ep` | The expert parallelism size. Default equal to tp size. | `tp-size` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--moe-a2a-backend` | Select the backend for all-to-all communication for expert parallelism. | `none` | `none`, `deepep`, `ascend_fuseep` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--moe-runner-backend` | Choose the runner backend for MoE. | `auto` | `auto` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--flashinfer-mxfp4-moe-precision` | Choose the computation precision of flashinfer mxfp4 moe | `default` | `default`, `bf16` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-flashinfer-allreduce-fusion` | Enable FlashInfer allreduce fusion with Residual RMSNorm. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--deepep-mode` | Select the mode when enable DeepEP MoE, could be `normal`, `low_latency` or `auto`. Default is `auto`, which means `low_latency` for decode batch and `normal` for prefill batch. | `auto` | `normal`, `low_latency`, `auto` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--ep-num-redundant-experts` | Allocate this number of redundant experts in expert parallel. | `0` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--ep-dispatch-algorithm` | The algorithm to choose ranks for redundant experts in expert parallel. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--init-expert-location` | Initial location of EP experts. | `trivial` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-eplb` | Enable EPLB algorithm | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--eplb-algorithm` | Chosen EPLB algorithm | `auto` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--eplb-rebalance-layers-per-chunk` | Number of layers to rebalance per forward pass. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--eplb-min-rebalancing-utilization-threshold` | Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range [0.0, 1.0]. | `1.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--expert-distribution-recorder-mode` | Mode of expert distribution recorder. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--expert-distribution-recorder-buffer-size` | Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-expert-distribution-metrics` | Enable logging metrics for expert balancedness | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--deepep-config` | Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--moe-dense-tp-size` | TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--elastic-ep-backend` | Select the collective communication backend for elastic EP. Currently supports 'mooncake'. | None | N/A | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mooncake-ib-device` | The InfiniBand devices for Mooncake Backend, accepts multiple comma-separated devices. Default is None, which triggers automatic device detection when Mooncake Backend is enabled. | None | N/A | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Mamba Cache
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|-------------------------------------|:--------------------------------------:|:--------------------------------------:|
| `--max-mamba-cache-size` | The maximum size of the mamba cache. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-ssm-dtype` | The data type of the SSM states in mamba cache. | `float32` | `float32`, `bfloat16` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-full-memory-ratio` | The ratio of mamba state memory to full kv cache memory. | `0.2` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-scheduler-strategy` | The strategy to use for mamba scheduler. `auto` currently defaults to `no_buffer`. 1. `no_buffer` does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. `extra_buffer` supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes `2x` for non-spec; `1+(1/(2+speculative_num_draft_tokens))x` for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)). 2a. `extra_buffer` is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support (currently only FLA backend), currently only extra_buffer supports branching. | `auto` | `auto`, `no_buffer`, `extra_buffer` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-track-interval` | The interval (in tokens) to track the mamba state during decode. Only used when `--mamba-scheduler-strategy` is `extra_buffer`. Must be divisible by page_size if set, and must be >= speculative_num_draft_tokens when using speculative decoding. | `256` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Args for multi-item scoring
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
| `--multi-item-scoring-delimiter` | Delimiter token ID for multi-item scoring. Used to combine Query and Items into a single sequence: Query<delimiter>Item1<delimiter>Item2<delimiter>... This enables efficient batch processing of multiple items against a single query. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Hierarchical cache
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--enable-hierarchical-cache` | Enable hierarchical cache | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-ratio` | The ratio of the size of host KV cache memory pool to the size of device pool. | `2.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-size` | The size of host KV cache memory pool in gigabytes, which will override the hicache_ratio if set. | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-write-policy` | The write policy of hierarchical cache. | `write_through` | `write_back`, `write_through`, `write_through_selective` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--radix-eviction-policy` | The eviction policy of radix trees. 'lru' stands for Least Recently Used, 'lfu' stands for Least Frequently Used. | `lru` | `lru`, `lfu` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-io-backend` | The IO backend for KV cache transfer between CPU and GPU | `kernel` | `kernel_ascend` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-mem-layout` | The layout of host memory pool for hierarchical cache. | `layer_first` | `page_first_direct`, `page_first_kv_split` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-storage-backend` | The storage backend for hierarchical KV cache. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: backend_name (custom name), module_path (Python module path), class_name (backend class name). | `None` | `file` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-storage-prefetch-policy` | Control when prefetching from the storage backend should stop. | `best_effort` | `best_effort`, `wait_complete`, `timeout` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--hicache-storage-backend-extra-config` | A dictionary in JSON string format containing extra configuration for the storage backend. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## LMCache
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|--------------------|-------------------------------------------------------------|----------|---------------------------|:--------------------------------------:|:--------------------------------------:|
| `--enable-lmcache` | Using LMCache as an alternative hierarchical cache solution | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Double Sparsity
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:--------------------------------------:|:--------------------------------------:|
| `--enable-double-sparsity` | Enable double sparsity attention | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--ds-channel-config-path` | The path of the double sparsity channel config | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--ds-heavy-channel-num` | The number of heavy channels in double sparsity attention | `32` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--ds-heavy-token-num` | The number of heavy tokens in double sparsity attention | `256` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--ds-heavy-channel-type` | The type of heavy channels in double sparsity attention | `qk` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--ds-sparse-decode-threshold` | The minimum decode sequence length required before the double-sparsity backend switches from the dense fallback to the sparse decode kernel. | `4096` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Offloading
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------|----------------------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
| `--cpu-offload-gb` | How many GBs of RAM to reserve for CPU offloading. | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--offload-group-size` | Number of layers per group in offloading. | `-1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--offload-num-in-group` | Number of layers to be offloaded within a group. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--offload-prefetch-step` | Steps to prefetch in offloading. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--offload-mode` | Mode of offloading. | `cpu` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Optimization/debug options
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|--------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--disable-radix-cache` | Disable RadixAttention for prefix caching. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--cuda-graph-max-bs` | Set the maximum batch size for cuda graph. It will extend the cuda graph capture batch size to this value. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--cuda-graph-bs` | Set the list of batch sizes for cuda graph. | `None` | List[int] | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--disable-cuda-graph` | Disable cuda graph. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disable-cuda-graph-padding` | Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--enable-profile-cuda-graph` | Enable profiling of cuda graph capture. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-cudagraph-gc` | Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen during capture to speed up the process. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-nccl-nvls` | Enable NCCL NVLS for prefill heavy requests when available. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-symm-mem` | Enable NCCL symmetric memory for fast collectives. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-flashinfer-cutlass-moe-fp4-allgather` | Disables quantize before all-gather for flashinfer cutlass moe. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-tokenizer-batch-encode` | Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input_ids, or input_embeds. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disable-outlines-disk-cache` | Disable disk cache of outlines to avoid possible crashes related to file system or high concurrency. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disable-custom-all-reduce` | Disable the custom all-reduce kernel and fall back to NCCL. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-mscclpp` | Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-torch-symm-mem` | Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM10 supports world size 6, 8. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-overlap-schedule` | Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-mixed-chunk` | Enabling mixing prefill and decode in a batch when using chunked prefill. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-dp-attention` | Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size. Currently DeepSeek-V2 and Qwen 2/3 MoE models are supported. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-dp-lm-head` | Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-two-batch-overlap` | Enabling two micro batches to overlap. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-single-batch-overlap` | Let computation and communication overlap within one micro batch. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--tbo-token-distribution-threshold` | The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap. | `0.48` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-torch-compile` | Optimize the model with torch.compile. Experimental feature. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-torch-compile-debug-mode` | Enable debug mode for torch compile. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-piecewise-cuda-graph` | Optimize the model with piecewise cuda graph for extend/prefill only. Experimental feature. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--piecewise-cuda-graph-tokens` | Set the list of tokens when using piecewise cuda graph. | `None` | Type: JSON list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--torch-compile-max-bs` | Set the maximum batch size when using torch compile. | `32` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--piecewise-cuda-graph-max-tokens` | Set the maximum tokens when using piecewise cuda graph. | `4096` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--torchao-config` | Optimize the model with torchao. Experimental feature. Current choices are: int8dq, int8wo, int4wo-<group_size>, fp8wo, fp8dq-per_tensor, fp8dq-per_row | `` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-nan-detection` | Enable the NaN detection for debugging purposes. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-p2p-check` | Enable P2P check for GPU access, otherwise the p2p access is allowed by default. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--triton-attention-reduce-in-fp32` | Cast the intermediate attention results to fp32 to avoid possible crashes related to fp16. This only affects Triton attention kernels. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--triton-attention-num-kv-splits` | The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8. | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--triton-attention-split-tile-size` | The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--num-continuous-decode-steps` | Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--delete-ckpt-after-loading` | Delete the model checkpoint after loading the model. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-memory-saver` | Allow saving memory using release_memory_occupation and resume_memory_occupation | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-weights-cpu-backup` | Save model weights to CPU memory during release_weights_occupation and resume_weights_occupation | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--allow-auto-truncate` | Allow automatically truncating requests that exceed the maximum input length instead of returning an error. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-custom-logit-processor` | Enable users to pass custom logit processors to the server (disabled by default for security) | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--flashinfer-mla-disable-ragged` | Not using ragged prefill wrapper when running flashinfer mla | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-shared-experts-fusion` | Disable shared experts fusion optimization for deepseek v3/r1. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-chunked-prefix-cache` | Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-fast-image-processor` | Adopt base image processor instead of fast image processor. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--keep-mm-feature-on-device` | Keep multimodal feature tensors on device after processing to save D2H copy. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-return-hidden-states` | Enable returning hidden states with responses. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--scheduler-recv-interval` | The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--numa-node` | Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess. | `None` | List[int] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-layerwise-nvtx-marker` | Enable layerwise NVTX profiling annotations for the model. This adds NVTX markers to every layer for detailed per-layer performance analysis with Nsight Systems. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-attn-tp-input-scattered` | Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-nsa-prefill-context-parallel` | Context parallelism used in the long sequence prefill phase of DeepSeek v3.2 | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Forward hooks
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-----------------|:--------------------------------------:|:--------------------------------------:|
| `--forward-hooks` | JSON-formatted list of forward hook specifications. Each element must include `target_modules` (list of glob patterns matched against `model.named_modules()` names) and `hook_factory` (Python import path to a factory, e.g. `my_package.hooks:make_hook`). An optional `name` field is used for logging, and an optional `config` object is passed as a `dict` to the factory. | `None` | Type: JSON list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Debug tensor dumps
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-------------------------------------------|---------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--debug-tensor-dump-input-file` | The input filename for dumping tensors | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--debug-tensor-dump-inject` | Inject the outputs from jax as the input of every layer. | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-dynamic-batch-tokenizer` | Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dynamic-batch-tokenizer-batch-size` | [Only used if --enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer. | `32` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dynamic-batch-tokenizer-batch-timeout` | [Only used if --enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests. | `0.002` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## PD disaggregation
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|--------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|--------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--disaggregation-mode` | Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated | `null` | `null`, `prefill`, `decode` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disaggregation-transfer-backend` | The backend for disaggregation transfer. Default is mooncake. | `mooncake` | `mooncake`, `nixl`, `ascend`, `fake` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disaggregation-bootstrap-port` | Bootstrap server port on the prefill server. Default is 8998. | `8998` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disaggregation-decode-tp` | Decode tp size. If not set, it matches the tp size of the current engine. This is only set on the prefill server. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disaggregation-decode-dp` | Decode dp size. If not set, it matches the dp size of the current engine. This is only set on the prefill server. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disaggregation-prefill-pp` | Prefill pp size. If not set, it is default to 1. This is only set on the decode server. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disaggregation-ib-device` | The InfiniBand devices for disaggregation transfer, accepts single device (e.g., --disaggregation-ib-device mlx5_0) or multiple comma-separated devices (e.g., --disaggregation-ib-device mlx5_0,mlx5_1). Default is None, which triggers automatic device detection when mooncake backend is enabled. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disaggregation-decode-enable-offload-kvcache` | Enable async KV cache offloading on decode server (PD mode). | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--num-reserved-decode-tokens` | Number of decode tokens that will have memory reserved when adding new request to the running batch. | `512` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disaggregation-decode-polling-interval` | The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Custom weight loader
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|------------------------------------------|
| `--custom-weight-loader` | The custom dataloader which used to update the model. Should be set with a valid import path, such as my_package.weight_load_func | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--weight-loader-disable-mmap` | Disable mmap while loading weight using safetensors. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--remote-instance-weight-loader-seed-instance-ip` | The ip of the seed instance for loading weights from remote instance. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-loader-seed-instance-service-port` | The service port of the seed instance for loading weights from remote instance. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-loader-send-weights-group-ports` | The communication group ports for loading weights from remote instance. | `None` | Type: JSON list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## For PD-Multiplexing
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-----------------------|--------------------------------------------------------|----------|---------------------------|:--------------------------------------:|:--------------------------------------:|
| `--enable-pdmux` | Enable PD-Multiplexing, PD running on greenctx stream. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--pdmux-config-path` | The path of the PD-Multiplexing config file. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--sm-group-num` | Number of sm partition groups. | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## For deterministic inference
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|------------------------------------|---------------------------------------------------------------|----------|---------------------------|----------------------------------------|----------------------------------------|
| `--enable-deterministic-inference` | Enable deterministic inference mode with batch invariant ops. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |

83
docs/platforms/ascend_npu_support_models.md Normal file
View File

@@ -0,0 +1,83 @@
# Support Models on Ascend NPU
This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language
Models, Embedding Models, and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included. You are welcome to enable
various models based on your business requirements.
## Large Language Models
| Model Family | Recommend Models | A2 Supported | A3 Supported |
|--------------------------------|--------------------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| DeepSeek | DeepSeek V1, V2, V3(V3.1,V3.2), R1 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen | Qwen 3, Qwen 3Moe | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Llama | meta-llama/Llama-4-Scout-17B-16E-Instruct,<br>AI-ModelScope/Llama-3.1-8B-Instruct,<br>LLM-Research/Llama-3.2-1B-Instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Mistral | mistralai/Mistral-7B-Instruct-v0.2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Gemma | google/gemma-3-4b-it | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Phi | microsoft/Phi-4-multimodal-instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| OLMoE | allenai/OLMoE-1B-7B-0924 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| StableLM | stabilityai/stablelm-2-1_6b | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Command-R | CohereForAI/c4ai-command-r-v01 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Grok | huihui-ai/grok-2 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| ChatGLM | ZhipuAI/chatglm2-6b | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| InternLM 2 | Shanghai_AI_Laboratory/internlm2-7b | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ExaONE 3 | LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| XVERSE | xverse/XVERSE-MoE-A36B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| SmolLM | HuggingFaceTB/SmolLM-1.7B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| GLM-4 | ZhipuAI/glm-4-9b-chat | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| MiMo | XiaomiMiMo/MiMo-7B-RL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Arcee AFM-4.5B | arcee-ai/AFM-4.5B-Base | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Persimmon | Howeee/persimmon-8b-chat | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Ling | inclusionAI/Ling-lite | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Granite | ibm-granite/granite-3.1-8b-instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Granite Moe | ibm-granite/granite-3.0-3b-a800m-instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| DBRX (Databricks) | databricks/dbrx-instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Baichuan 2 (7B, 13B) | baichuan-inc/Baichuan2-13B-Chat | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| ERNIE-4.5 (4.5, 4.5MoE series) | baidu/ERNIE-4.5-21B-A3B-PT | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| MiniCPM (v3, 4B) | openbmb/MiniCPM3-4B | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| GPTOSS | openai/gpt-oss-120b | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Multimodal Language Models
| Model Family | Recommend Models | A2 Supported | A3 Supported |
|--------------------------------|-----------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| Qwen-VL (Qwen2 series) | Qwen/Qwen3-VL-235B-A22B-Instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| DeepSeek-VL2 | deepseek-ai/deepseek-vl2 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Janus-Pro (1B, 7B) | deepseek-ai/Janus-Pro-7B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| MiniCPM-V / MiniCPM-o | openbmb/MiniCPM-V-2_6 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Gemma 3 (Multimodal) | google/gemma-3-4b-it | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Mistral-Small-3.1-24B | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Phi-4-multimodal-instruct | microsoft/Phi-4-multimodal-instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| MiMo-VL (7B) | XiaomiMiMo/MiMo-VL-7B-RL | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| LLaVA (v1.5 & v1.6) | AI-ModelScope/llava-v1.6-34b | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| LLaVA-NeXT (8B, 72B) | lmms-lab/llava-next-72b | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| LLaVA-OneVision | lmms-lab/llava-onevision-qwen2-7b-ov | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Kimi-VL (A3B) | Kimi/Kimi-VL-A3B-Instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| GLM-4.5V (106B) / GLM-4.1V(9B) | ZhipuAI/GLM-4.5V | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| Llama 3.2 Vision (11B) | meta-llama/Llama-3.2-11B-Vision-Instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Embedding Models
| Model Family | Recommend Models | A2 Supported | A3 Supported |
|--------------------------|------------------------------------------|:--------------------------------------:|:----------------------------------------:|
| E5 (Llama/Mistral based) | intfloat/e5-mistral-7b-instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| GTE-Qwen2 | iic/gte_Qwen2-1.5B-instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Qwen3-Embedding | Qwen/Qwen3-Embedding-8B | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| GME (Multimodal) | Alibaba-NLP/gme-Qwen2-VL-2B-Instruct | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| CLIP | AI-ModelScope/clip-vit-large-patch14-336 | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| BGE | BAAI/bge-large-en-v1.5 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Reward Models
| Model Family | Recommend Models | A2 Supported | A3 Supported |
|---------------------------|--------------------------------------------|:--------------------------------------:|:----------------------------------------:|
| Llama3.1 Reward | Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| InternLM 2 Reward | Shanghai_AI_Laboratory/internlm2-7b-reward | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| Qwen2.5 Reward - Math | Qwen/Qwen2.5-Math-RM-72B | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| Qwen2.5 Reward - Sequence | jason9693/Qwen2.5-1.5B-apeach | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| Gemma 2-27B Reward | Skywork/Skywork-Reward-Gemma-2-27B-v0.2 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Rerank Models
| Model Family | Recommend Models | A2 Supported | A3 Supported |
|--------------|-------------------------|:--------------------------------------:|:--------------------------------------:|
| BGE-Reranker | BAAI/bge-reranker-v2-m3 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |