[npu] update model and feature supported for ascend npu (#16390)
This commit is contained in:
Hexq0210
@@ -3,12 +3,13 @@
|
||||
This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any
|
||||
questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
|
||||
|
||||
If you want to know the meaning and usage of each parameter, click [Service Arguments](https://docs.sglang.io/advanced_features/server_arguments.html).
|
||||
If you want to know the meaning and usage of each parameter,
|
||||
click [Service Arguments](https://docs.sglang.io/advanced_features/server_arguments.html).
|
||||
|
||||
## Model and tokenizer
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|----------------------------------------|----------|---------------------------------------|------------------------------------------|------------------------------------------|
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|----------------------------------------|----------|---------------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--model-path`<br/>`--model` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tokenizer-path` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tokenizer-mode` | `auto` | `auto`, `slow` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -20,27 +21,37 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--context-length` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--is-embedding` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-multimodal` | `None` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--revision` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--model-impl` | `None` | `auto`, `sglang`,<br/> `transformers` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--revision` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--model-impl` | `None` | `auto`, `sglang`,<br/> `transformers` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## HTTP server
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|------------------------|-------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--host` | `127.0.0.1` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--port` | `30000` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--skip-server-warmup` | `False` | bool flag <br/>(set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--warmups` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--nccl-port` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------------------------------------------------|-------------|---------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--host` | `127.0.0.1` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--port` | `30000` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--skip-server-warmup` | `False` | bool flag <br/>(set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--warmups` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--nccl-port` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--fastapi-root-path` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--grpc-mode` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--checkpoint-engine-` <br/> `wait-weights-` <br/> `before-ready` | FALSE | bool flag <br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Quantization and data type
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-----------------------------|----------|-----------------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--dtype` | `auto` | `auto`,<br/> `float16`,<br/> `bfloat16` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--quantization` | `None` | `modelslim` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--quantization-param-path` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--kv-cache-dtype` | `auto` | `auto` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------------------------------|----------|-----------------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--dtype` | `auto` | `auto`,<br/> `float16`,<br/> `bfloat16` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--quantization` | `None` | `modelslim` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--quantization-param-path` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--kv-cache-dtype` | `auto` | `auto` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-fp32-lm-head` | FALSE | bool flag <br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--modelopt-quant` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--modelopt-checkpoint-`<br/>`restore-path` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--modelopt-checkpoint-`<br/>`save-path` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--modelopt-export-path` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--quantize-and-serve--`<br/>`rl-quant-profile` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--load-format flash_rl` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Memory and scheduling
|
||||
|
||||
@@ -62,6 +73,8 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--hybrid-kvcache-ratio` | `None` | Optional[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--swa-full-tokens-ratio` | `0.8` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-hybrid-swa-memory` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--abort-on-priority-`<br/>`when-disabled` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-dynamic-chunking` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Runtime options
|
||||
|
||||
@@ -71,6 +84,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--pp-max-micro-batch-size` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--pp-async-batch-depth` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--stream-interval` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--stream-output` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--random-seed` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -78,11 +92,11 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--constrained-json-`<br/>`disable-any-whitespace` | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--watchdog-timeout` | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--soft-watchdog-timeout` | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--dist-timeout` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--dist-timeout` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--base-gpu-id` | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--gpu-id-step` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--sleep-on-idle` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--mm-process-config` | `{}` | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--custom-sigquit-handler` | `{}` | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Logging
|
||||
|
||||
@@ -92,6 +106,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--log-level-http` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--log-requests` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--log-requests-format` | text | text, json | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--crash-dump-folder` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--crash-on-nan` | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-metrics` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
@@ -106,11 +121,18 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--generation-tokens-buckets` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--gc-warning-threshold-secs` | `0.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--decode-log-interval` | `40` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-request-time-`<br/>`stats-logging` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-request-time-`<br/>`stats-logging` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--kv-events-config` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-trace` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--oltp-traces-endpoint` | `localhost:4317` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## RequestMetricsExporter configuration
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|---------------------------------------|----------|---------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--export-metrics-to-`<br/>`file` | FALSE | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--export-metrics-to-`<br/>`file-dir` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## API related
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
@@ -132,6 +154,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
|----------------------------------------|---------------|-------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--data-parallel-size`<br/>`--dp-size` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--load-balance-method` | `round_robin` | `round_robin`,<br/> `shortest_queue`,<br/> `minimum_tokens` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--prefill-round-robin-balance` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Multi-node distributed serving
|
||||
|
||||
@@ -154,12 +177,12 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
|--------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--enable-lora` | `False` | Bool flag <br/>(set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--max-lora-rank` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--lora-target-modules` | `None` | `q_proj`, <br/>`k_proj`,<br/> `v_proj`,<br/> `o_proj`,<br/> `gate_proj`,<br/> `up_proj`,<br/> `down_proj`,<br/> `qkv_proj`,<br/> `gate_up_proj`,<br/> `all` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--lora-target-modules` | `None` | `q_proj`, <br/>`k_proj`,<br/> `v_proj`,<br/> `o_proj`,<br/> `gate_proj`,<br/> `up_proj`,<br/> `down_proj`,<br/> `qkv_proj`,<br/> `gate_up_proj`,<br/> `all` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--lora-paths` | `None` | Type: List[str] /<br/> JSON objects | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--max-loras-per-batch` | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--max-loaded-loras` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--lora-eviction-policy` | `lru` | `lru`,<br/> `fifo` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--lora-backend` | `triton` | `triton`,<br/> `csgmv` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--max-loras-per-batch` | `8` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--max-loaded-loras` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--lora-eviction-policy` | `lru` | `lru`,<br/> `fifo` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--lora-backend` | `triton` | `triton`,<br/> `csgmv` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--max-lora-chunk-size` | `16` | `16`, `32`,<br/> `64`, `128` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Kernel Backends (Attention, Sampling, Grammar, GEMM)
|
||||
@@ -184,6 +207,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--speculative-algorithm` | `None` | `EAGLE`,<br/> `EAGLE3`,<br/> `NEXTN`,<br/> `STANDALONE`, <br/>`NGRAM` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-draft-model-path`<br/>`--speculative-draft-model` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-draft-model-`<br/>`revision` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--speculative-draft-load-format` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--speculative-num-steps` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-eagle-topk` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-num-draft-tokens` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -193,6 +217,8 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--speculative-attention-`<br/>`mode` | `prefill` | `prefill`,<br/> `decode` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-moe-runner-`<br/>`backend` | `None` | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-moe-a2a-`<br/>`backend` | `None` | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-draft-attention-backend` | | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-draft-model-quantization` | | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Ngram speculative decoding
|
||||
|
||||
@@ -241,12 +267,6 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--mamba-scheduler-strategy` | `auto` | `auto`, <br/>`no_buffer`,<br/> `extra_buffer` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--mamba-track-interval` | `256` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Args for multi-item scoring
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|----------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--multi-item-scoring-delimiter` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Hierarchical cache
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
@@ -268,6 +288,17 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
|--------------------|----------|--------------------------------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--enable-lmcache` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Ktransformer server args
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|----------------------------------------------|----------|---------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--kt-weight-path` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--kt-method` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--kt-cpuinfer` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--kt-threadpool-count` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--kt-num-gpu-experts` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--kt-max-deferred-`<br/>`experts-per-token` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Double Sparsity
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
@@ -289,6 +320,12 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--offload-prefetch-step` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--offload-mode` | `cpu` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Args for multi-item scoring
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|----------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--multi-item-scoring-delimiter` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Optimization/debug options
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
@@ -304,6 +341,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--enable-symm-mem` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-flashinfer-`<br/>`cutlass-moe-fp4-allgather` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-outlines-`<br/>`disk-cache` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disable-custom-`<br/>`all-reduce` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-mscclpp` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
@@ -319,6 +357,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--enable-torch-`<br/>`compile-debug-mode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-piecewise-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--piecewise-cuda-`<br/>`graph-tokens` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--piecewise-cuda-`<br/>`graph-compiler` | `eager` | ["eager", "inductor"] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--torch-compile-max-bs` | `32` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--piecewise-cuda-`<br/>`graph-max-tokens` | `4096` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--torchao-config` | `` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
@@ -331,6 +370,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-memory-saver` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-draft-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
@@ -339,28 +379,32 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-return-`<br/>`routed-experts` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--scheduler-recv-`<br/>`interval` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--numa-node` | `None` | List[int] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--rl-on-policy-target` | `None` | `fsdp` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-layerwise-`<br/>`nvtx-marker` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-attn-tp-`<br/>`input-scattered` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-nsa-prefill-`<br/>`context-parallel` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-fused-qk-`<br/>`norm-rope` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Forward hooks
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------|----------|-----------------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--forward-hooks` | `None` | Type: JSON list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Debug tensor dumps
|
||||
## Dynamic batch tokenizer
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|--------------------------------------------------|----------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--debug-tensor-dump-`<br/>`input-file` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--debug-tensor-dump-`<br/>`inject` | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-dynamic-`<br/>`batch-tokenizer` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--dynamic-batch-`<br/>`tokenizer-batch-size` | `32` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--dynamic-batch-`<br/>`tokenizer-batch-timeout` | `0.002` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Debug tensor dumps
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|--------------------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--debug-tensor-dump-`<br/>`output-folder` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--debug-tensor-dump-`<br/>`layers` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--debug-tensor-dump-`<br/>`input-file` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--debug-tensor-dump-`<br/>`inject` | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## PD disaggregation
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
@@ -373,18 +417,30 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--disaggregation-prefill-pp` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disaggregation-ib-device` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disaggregation-decode-`<br/>`enable-offload-kvcache` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disaggregation-decode-`<br/>`enable-fake-auto` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--num-reserved-decode-tokens` | `512` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disaggregation-decode-`<br/>`polling-interval` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Encode prefill disaggregation
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|------------------------------|----------|---------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--encoder-only` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--language-only` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--encoder-transfer-backend` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--encoder-urls` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Custom weight loader
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|---------------------------------------------------------------------|----------|--------------------------------|:----------------------------------------:|------------------------------------------|
|
||||
| `--custom-weight-loader` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------------------------------------------------------|----------|--------------------------------|:----------------------------------------:|------------------------------------------|
|
||||
| `--custom-weight-loader` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-backend` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-start-seed-via-transfer-engine` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## For PD-Multiplexing
|
||||
|
||||
@@ -394,8 +450,39 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|
||||
| `--pdmux-config-path` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--sm-group-num` | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## For Multi-Modal
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-----------------------------------------------|----------|-------------------|------------------------------------------|------------------------------------------|
|
||||
| `--mm-max-concurrent-calls` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--mm-per-request-timeout` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-broadcast-mm-`<br/>`inputs-process` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--mm-process-config` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--mm-enable-dp-encoder` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--limit-mm-data-per-request` | `None` | Type: JSON / Dict | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## For checkpoint decryption
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|---------------------------------|----------|---------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--decrypted-config-file` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--decrypted-draft-config-file` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-prefix-mm-cache` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## For deterministic inference
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------------------------|----------|--------------------------------|----------------------------------------|----------------------------------------|
|
||||
| `--enable-deterministic-`<br/>`inference` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## For registering hooks
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------|----------|-----------------|----------------------------------------|----------------------------------------|
|
||||
| `--forward-hooks` | `None` | Type: JSON list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Configuration file support
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|------------|----------|---------|----------------------------------------|----------------------------------------|
|
||||
| `--config` | | yaml | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
@@ -9,7 +9,7 @@ You are welcome to enable various models based on your business requirements.
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|-------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| DeepSeek V3/V3.1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek V3.2 | DeepSeek | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek V3.2 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek R1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek V2 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -26,9 +26,9 @@ You are welcome to enable various models based on your business requirements.
|
||||
| microsoft/Phi-4-multimodal-instruct | Phi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| allenai/OLMoE-1B-7B-0924 | OLMoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| stabilityai/stablelm-2-1_6b | StableLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| CohereForAI/c4ai-command-r-v01 | Command-R | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| CohereForAI/c4ai-command-r-v01 | Command-R | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| huihui-ai/grok-2 | Grok | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/chatglm2-6b | ChatGLM | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/chatglm2-6b | ChatGLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| xverse/XVERSE-MoE-A36B | XVERSE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -50,45 +50,45 @@ You are welcome to enable various models based on your business requirements.
|
||||
|
||||
| Models | Model Family (Variants) | A2 Supported | A3 Supported |
|
||||
|-----------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:|
|
||||
| Qwen2.5-VL-72B-Instruct-w8a8 | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-VL-8B-Instruct | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen2.5-VL-72B-Instruct-w8a8 | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-VL-8B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| google/gemma-3-4b-it | Gemma 3 (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.1-24B | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| microsoft/Phi-4-multimodal-instruct | Phi-4-multimodal-instruct | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| XiaomiMiMo/MiMo-VL-7B-RL | MiMo-VL (7B) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.1-24B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| microsoft/Phi-4-multimodal-instruct | Phi-4-multimodal-instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| XiaomiMiMo/MiMo-VL-7B-RL | MiMo-VL (7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/llava-v1.6-34b | LLaVA (v1.5 & v1.6) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| lmms-lab/llava-next-72b | LLaVA-NeXT (8B, 72B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| lmms-lab/llava-onevision-qwen2-7b-ov | LLaVA-OneVision | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Kimi/Kimi-VL-A3B-Instruct | Kimi-VL (A3B) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/GLM-4.5V | GLM-4.5V (106B) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| lmms-lab/llava-onevision-qwen2-7b-ov | LLaVA-OneVision | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Kimi/Kimi-VL-A3B-Instruct | Kimi-VL (A3B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/GLM-4.5V | GLM-4.5V (106B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| meta-llama/Llama-3.2-11B-Vision-Instruct | Llama 3.2 Vision (11B) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Embedding Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|-------------------------------------------|--------------------------|------------------------------------------|:----------------------------------------:|
|
||||
| intfloat/e5-mistral-7b-instruct | E5 (Llama/Mistral based) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| iic/gte_Qwen2-1.5B-instruct | GTE-Qwen2 | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| intfloat/e5-mistral-7b-instruct | E5 (Llama/Mistral based) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| iic/gte_Qwen2-1.5B-instruct | GTE-Qwen2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-Embedding-8B | Qwen3-Embedding | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| Alibaba-NLP/gme-Qwen2-VL-2B-Instruct | GME (Multimodal) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Alibaba-NLP/gme-Qwen2-VL-2B-Instruct | GME (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/clip-vit-large-patch14-336 | CLIP | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| BAAI/bge-large-en-v1.5 | BGE | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Reward Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|---------------------------------------------|---------------------------|----------------------------------------|:----------------------------------------:|
|
||||
| Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| jason9693/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| Skywork/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|---------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:|
|
||||
| Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| jason9693/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Skywork/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Rerank Models
|
||||
|
||||
|
||||
Reference in New Issue
Block a user