From e834b85ab677971ffcdf60edb373f4472fe37224 Mon Sep 17 00:00:00 2001 From: Hexq0210 <893781835@qq.com> Date: Sat, 7 Feb 2026 16:38:05 +0800 Subject: [PATCH] [NPU] update npu doc (#18344) --- docs/platforms/ascend_npu_support_features.md | 487 ++++++++++++++++++ docs/platforms/ascend_npu_support_models.md | 20 +- 2 files changed, 500 insertions(+), 7 deletions(-) create mode 100644 docs/platforms/ascend_npu_support_features.md diff --git a/docs/platforms/ascend_npu_support_features.md b/docs/platforms/ascend_npu_support_features.md new file mode 100644 index 000000000..0c1cb379b --- /dev/null +++ b/docs/platforms/ascend_npu_support_features.md @@ -0,0 +1,487 @@ +# Support Features on Ascend NPU + +This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any +questions, please [open an issue](https://github.com/sgl-project/sglang/issues). + +If you want to know the meaning and usage of each parameter, +click [Server Arguments](https://docs.sglang.io/advanced_features/server_arguments.html). + +## Model and tokenizer + +| Argument | Defaults | Options | Server supported | +|----------------------------------------|----------|---------------------------------------|:----------------:| +| `--model-path`
`--model` | `None` | Type: str | A2, A3 | +| `--tokenizer-path` | `None` | Type: str | A2, A3 | +| `--tokenizer-mode` | `auto` | `auto`, `slow` | A2, A3 | +| `--tokenizer-worker-num` | `1` | Type: int | A2, A3 | +| `--skip-tokenizer-init` | `False` | bool flag (set to enable) | A2, A3 | +| `--load-format` | `auto` | `auto`, `safetensors` | A2, A3 | +| `--model-loader-`
`extra-config` | {} | Type: str | A2, A3 | +| `--trust-remote-code` | `False` | bool flag (set to enable) | A2, A3 | +| `--context-length` | `None` | Type: int | A2, A3 | +| `--is-embedding` | `False` | bool flag (set to enable) | A2, A3 | +| `--enable-multimodal` | `None` | bool flag (set to enable) | A2, A3 | +| `--revision` | `None` | Type: str | A2, A3 | +| `--model-impl` | `auto` | `auto`, `sglang`,
`transformers` | A2, A3 | + +## HTTP server + +| Argument | Defaults | Options | Server supported | +|------------------------|-------------|---------------------------|:----------------:| +| `--host` | `127.0.0.1` | Type: str | A2, A3 | +| `--port` | `30000` | Type: int | A2, A3 | +| `--skip-server-warmup` | `False` | bool flag (set to enable) | A2, A3 | +| `--warmups` | `None` | Type: str | A2, A3 | +| `--nccl-port` | `None` | Type: int | A2, A3 | +| `--fastapi-root-path` | `None` | Type: str | A2, A3 | +| `--grpc-mode` | `False` | bool flag (set to enable) | A2, A3 | + +## Quantization and data type + +| Argument | Defaults | Options | Server supported | +|---------------------------------------------|----------|-----------------------------------------|:----------------:| +| `--dtype` | `auto` | `auto`,
`float16`,
`bfloat16` | A2, A3 | +| `--quantization` | `None` | `modelslim` | A2, A3 | +| `--quantization-param-path` | `None` | Type: str | Special For GPU | +| `--kv-cache-dtype` | `auto` | `auto` | A2, A3 | +| `--enable-fp32-lm-head` | `False` | bool flag
(set to enable) | A2, A3 | +| `--modelopt-quant` | `None` | Type: str | Special For GPU | +| `--modelopt-checkpoint-`
`restore-path` | `None` | Type: str | Special For GPU | +| `--modelopt-checkpoint-`
`save-path` | `None` | Type: str | Special For GPU | +| `--modelopt-export-path` | `None` | Type: str | Special For GPU | +| `--quantize-and-serve` | `False` | bool flag
(set to enable) | Special For GPU | +| `--rl-quant-profile` | `None` | Type: str | Special For GPU | + +## Memory and scheduling + +| Argument | Defaults | Options | Server supported | +|-----------------------------------------------------|----------|--------------------------------|:----------------:| +| `--mem-fraction-static` | `None` | Type: float | A2, A3 | +| `--max-running-requests` | `None` | Type: int | A2, A3 | +| `--prefill-max-requests` | `None` | Type: int | A2, A3 | +| `--max-queued-requests` | `None` | Type: int | A2, A3 | +| `--max-total-tokens` | `None` | Type: int | A2, A3 | +| `--chunked-prefill-size` | `None` | Type: int | A2, A3 | +| `--max-prefill-tokens` | `16384` | Type: int | A2, A3 | +| `--schedule-policy` | `fcfs` | `lpm`, `fcfs` | A2, A3 | +| `--enable-priority-`
`scheduling` | `False` | bool flag
(set to enable) | A2, A3 | +| `--schedule-low-priority-`
`values-first` | `False` | bool flag
(set to enable) | A2, A3 | +| `--priority-scheduling-`
`preemption-threshold` | `10` | Type: int | A2, A3 | +| `--schedule-conservativeness` | `1.0` | Type: float | A2, A3 | +| `--page-size` | `128` | Type: int | A2, A3 | +| `--swa-full-tokens-ratio` | `0.8` | Type: float | A2, A3 | +| `--disable-hybrid-swa-memory` | `False` | bool flag
(set to enable) | A2, A3 | +| `--abort-on-priority-`
`when-disabled` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-dynamic-chunking` | `False` | bool flag
(set to enable) | A2, A3 | + +## Runtime options + +| Argument | Defaults | Options | Server supported | +|----------------------------------------------------|----------|---------------------------|:----------------:| +| `--device` | `None` | Type: str | A2, A3 | +| `--tensor-parallel-size`
`--tp-size` | `1` | Type: int | A2, A3 | +| `--pipeline-parallel-size`
`--pp-size` | `1` | Type: int | A2, A3 | +| `--pp-max-micro-batch-size` | `None` | Type: int | A2, A3 | +| `--pp-async-batch-depth` | `None` | Type: int | A2, A3 | +| `--stream-interval` | `1` | Type: int | A2, A3 | +| `--stream-output` | `False` | bool flag (set to enable) | A2, A3 | +| `--random-seed` | `None` | Type: int | A2, A3 | +| `--constrained-json-`
`whitespace-pattern` | `None` | Type: str | A2, A3 | +| `--constrained-json-`
`disable-any-whitespace` | `False` | bool flag (set to enable) | A2, A3 | +| `--watchdog-timeout` | `300` | Type: float | A2, A3 | +| `--soft-watchdog-timeout` | `300` | Type: float | A2, A3 | +| `--dist-timeout` | `None` | Type: int | A2, A3 | +| `--base-gpu-id` | `0` | Type: int | A2, A3 | +| `--gpu-id-step` | `1` | Type: int | A2, A3 | +| `--sleep-on-idle` | `False` | bool flag (set to enable) | A2, A3 | +| `--custom-sigquit-handler` | `None` | Optional[Callable] | A2, A3 | + +## Logging + +| Argument | Defaults | Options | Server supported | +|----------------------------------------------------|-------------------|--------------------------------|:----------------:| +| `--log-level` | `info` | Type: str | A2, A3 | +| `--log-level-http` | `None` | Type: str | A2, A3 | +| `--log-requests` | `False` | bool flag
(set to enable) | A2, A3 | +| `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | A2, A3 | +| `--log-requests-format` | text | text, json | A2, A3 | +| `--crash-dump-folder` | `None` | Type: str | A2, A3 | +| `--enable-metrics` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-metrics-for-`
`all-schedulers` | `False` | bool flag
(set to enable) | A2, A3 | +| `--tokenizer-metrics-`
`custom-labels-header` | `x-custom-labels` | Type: str | A2, A3 | +| `--tokenizer-metrics-`
`allowed-custom-labels` | `None` | List[str] | A2, A3 | +| `--bucket-time-to-`
`first-token` | `None` | List[float] | A2, A3 | +| `--bucket-inter-token-`
`latency` | `None` | List[float] | A2, A3 | +| `--bucket-e2e-request-`
`latency` | `None` | List[float] | A2, A3 | +| `--collect-tokens-`
`histogram` | `False` | bool flag
(set to enable) | A2, A3 | +| `--prompt-tokens-buckets` | `None` | List[str] | A2, A3 | +| `--generation-tokens-buckets` | `None` | List[str] | A2, A3 | +| `--gc-warning-threshold-secs` | `0.0` | Type: float | A2, A3 | +| `--decode-log-interval` | `40` | Type: int | A2, A3 | +| `--enable-request-time-`
`stats-logging` | `False` | bool flag
(set to enable) | A2, A3 | +| `--kv-events-config` | `None` | Type: str | Special for GPU | +| `--enable-trace` | `False` | bool flag
(set to enable) | A2, A3 | +| `--oltp-traces-endpoint` | `localhost:4317` | Type: str | A2, A3 | + +## RequestMetricsExporter configuration + +| Argument | Defaults | Options | Server supported | +|---------------------------------------|----------|--------------------------------|:----------------:| +| `--export-metrics-to-`
`file` | `False` | bool flag
(set to enable) | A2, A3 | +| `--export-metrics-to-`
`file-dir` | `None` | Type: str | A2, A3 | + +## API related + +| Argument | Defaults | Options | Server supported | +|-------------------------|-----------|--------------------------------|:----------------:| +| `--api-key` | `None` | Type: str | A2, A3 | +| `--admin-api-key` | `None` | Type: str | A2, A3 | +| `--served-model-name` | `None` | Type: str | A2, A3 | +| `--weight-version` | `default` | Type: str | A2, A3 | +| `--chat-template` | `None` | Type: str | A2, A3 | +| `--completion-template` | `None` | Type: str | A2, A3 | +| `--enable-cache-report` | `False` | bool flag
(set to enable) | A2, A3 | +| `--reasoning-parser` | `None` | `deepseek-r1` | A2, A3 | +| `--tool-call-parser` | `None` | `llama`,`pythonic` | A2, A3 | +| `--sampling-defaults` | `model` | `openai`, `model` | A2, A3 | + +## Data parallelism + +| Argument | Defaults | Options | Server supported | +|----------------------------------------|---------------|-----------------------------------------------------------|:----------------:| +| `--data-parallel-size`
`--dp-size` | `1` | Type: int | A2, A3 | +| `--load-balance-method` | `round_robin` | `round_robin`,
`total_requests`,
`total_tokens` | A2, A3 | +| `--prefill-round-robin-balance` | `False` | bool flag
(set to enable) | A2, A3 | + +## Multi-node distributed serving + +| Argument | Defaults | Options | Server supported | +|-------------------------------------------|----------|-----------|:----------------:| +| `--dist-init-addr`
`--nccl-init-addr` | `None` | Type: str | A2, A3 | +| `--nnodes` | `1` | Type: int | A2, A3 | +| `--node-rank` | `0` | Type: int | A2, A3 | + +## Model override args + +| Argument | Defaults | Options | Server supported | +|--------------------------------------|----------|-----------|:----------------:| +| `--json-model-override-`
`args` | `{}` | Type: str | A2, A3 | +| `--preferred-sampling-`
`params` | `None` | Type: str | A2, A3 | + +## LoRA + +| Argument | Defaults | Options | Server supported | +|--------------------------|----------|-------------------------------------|:----------------:| +| `--enable-lora` | `False` | Bool flag
(set to enable) | A2, A3 | +| `--max-lora-rank` | `None` | Type: int | A2, A3 | +| `--lora-target-modules` | `None` | `all` | A2, A3 | +| `--lora-paths` | `None` | Type: List[str] /
JSON objects | A2, A3 | +| `--max-loras-per-batch` | `8` | Type: int | A2, A3 | +| `--max-loaded-loras` | `None` | Type: int | A2, A3 | +| `--lora-eviction-policy` | `lru` | `lru`,
`fifo` | A2, A3 | +| `--lora-backend` | `triton` | `triton` | A2, A3 | +| `--max-lora-chunk-size` | `16` | `16`, `32`,
`64`, `128` | Special for GPU | + +## Kernel Backends (Attention, Sampling, Grammar, GEMM) + +| Argument | Defaults | Options | Server supported | +|----------------------------------------|-------------------|------------------------------------------------------------------------------------------------|:----------------:| +| `--attention-backend` | `None` | `ascend` | A2, A3 | +| `--prefill-attention-backend` | `None` | `ascend` | A2, A3 | +| `--decode-attention-backend` | `None` | `ascend` | A2, A3 | +| `--sampling-backend` | `None` | `pytorch`,
`ascend` | A2, A3 | +| `--grammar-backend` | `None` | `xgrammar` | A2, A3 | +| `--mm-attention-backend` | `None` | `ascend_attn` | A2, A3 | +| `--nsa-prefill-backend` | `flashmla_sparse` | `flashmla_sparse`,
`flashmla_decode`,
`fa3`,
`tilelang`,
`aiter` | Special for GPU | +| `--nsa-decode-backend` | `fa3` | `flashmla_prefill`,
`flashmla_kv`,
`fa3`,
`tilelang`,
`aiter` | Special for GPU | +| `--fp8-gemm-backend` | `auto` | `auto`,
`deep_gemm`,
`flashinfer_trtllm`,
`cutlass`,
`triton`,
`aiter` | Special for GPU | +| `--disable-flashinfer-`
`autotune` | `False` | bool flag
(set to enable) | Special for GPU | + +## Speculative decoding + +| Argument | Defaults | Options | Server supported | +|------------------------------------------------------------------|-----------|--------------------------|:----------------:| +| `--speculative-algorithm` | `None` | `EAGLE3`,
`NEXTN` | A2, A3 | +| `--speculative-draft-model-path`
`--speculative-draft-model` | `None` | Type: str | A2, A3 | +| `--speculative-draft-model-`
`revision` | `None` | Type: str | A2, A3 | +| `--speculative-draft-load-format` | `None` | `auto` | A2, A3 | +| `--speculative-num-steps` | `None` | Type: int | A2, A3 | +| `--speculative-eagle-topk` | `None` | Type: int | A2, A3 | +| `--speculative-num-draft-tokens` | `None` | Type: int | A2, A3 | +| `--speculative-accept-`
`threshold-single` | `1.0` | Type: float | Special for GPU | +| `--speculative-accept-`
`threshold-acc` | `1.0` | Type: float | Special for GPU | +| `--speculative-token-map` | `None` | Type: str | A2, A3 | +| `--speculative-attention-`
`mode` | `prefill` | `prefill`,
`decode` | A2, A3 | +| `--speculative-moe-runner-`
`backend` | `None` | `auto` | A2, A3 | +| `--speculative-moe-a2a-`
`backend` | `None` | `ascend_fuseep` | A2, A3 | +| `--speculative-draft-attention-backend` | `None` | `ascend` | A2, A3 | +| `--speculative-draft-model-quantization` | `None` | `unquant` | A2, A3 | + +## Ngram speculative decoding + +| Argument | Defaults | Options | Server supported | +|----------------------------------------------------|------------|--------------------|:----------------:| +| `--speculative-ngram-`
`min-match-window-size` | `1` | Type: int | Experimental | +| `--speculative-ngram-`
`max-match-window-size` | `12` | Type: int | Experimental | +| `--speculative-ngram-`
`min-bfs-breadth` | `1` | Type: int | Experimental | +| `--speculative-ngram-`
`max-bfs-breadth` | `10` | Type: int | Experimental | +| `--speculative-ngram-`
`match-type` | `BFS` | `BFS`,
`PROB` | Experimental | +| `--speculative-ngram-`
`branch-length` | `18` | Type: int | Experimental | +| `--speculative-ngram-`
`capacity` | `10000000` | Type: int | Experimental | + +## Expert parallelism + +| Argument | Defaults | Options | Server supported | +|-------------------------------------------------------|-----------|---------------------------------------------|:----------------:| +| `--expert-parallel-size`
`--ep-size`
`--ep` | `1` | Type: int | A2, A3 | +| `--moe-a2a-backend` | `none` | `none`,
`deepep`,
`ascend_fuseep` | A2, A3 | +| `--moe-runner-backend` | `auto` | `auto`, `triton` | A2, A3 | +| `--flashinfer-mxfp4-`
`moe-precision` | `default` | `default`,
`bf16` | Special for GPU | +| `--enable-flashinfer-`
`allreduce-fusion` | `False` | bool flag
(set to enable) | Special for GPU | +| `--deepep-mode` | `auto` | `normal`,
`low_latency`,
`auto` | A2, A3 | +| `--deepep-config` | `None` | Type: str | Special for GPU | +| `--ep-num-redundant-experts` | `0` | Type: int | A2, A3 | +| `--ep-dispatch-algorithm` | `None` | Type: str | A2, A3 | +| `--init-expert-location` | `trivial` | Type: str | A2, A3 | +| `--enable-eplb` | `False` | bool flag
(set to enable) | A2, A3 | +| `--eplb-algorithm` | `auto` | Type: str | A2, A3 | +| `--eplb-rebalance-layers-`
`per-chunk` | `None` | Type: int | A2, A3 | +| `--eplb-min-rebalancing-`
`utilization-threshold` | `1.0` | Type: float | A2, A3 | +| `--expert-distribution-`
`recorder-mode` | `None` | Type: str | A2, A3 | +| `--expert-distribution-`
`recorder-buffer-size` | `None` | Type: int | A2, A3 | +| `--enable-expert-distribution-`
`metrics` | `False` | bool flag (set to enable) | A2, A3 | +| `--moe-dense-tp-size` | `None` | Type: int | A2, A3 | +| `--elastic-ep-backend` | `None` | `none`, `mooncake` | Special for GPU | +| `--mooncake-ib-device` | `None` | Type: str | Special for GPU | + +## Mamba Cache + +| Argument | Defaults | Options | Server supported | +|------------------------------|-----------|-----------------------------------------------|:----------------:| +| `--max-mamba-cache-size` | `None` | Type: int | A2, A3 | +| `--mamba-ssm-dtype` | `float32` | `float32`,
`bfloat16` | A2, A3 | +| `--mamba-full-memory-ratio` | `0.2` | Type: float | A2, A3 | +| `--mamba-scheduler-strategy` | `auto` | `auto`,
`no_buffer`,
`extra_buffer` | A2, A3 | +| `--mamba-track-interval` | `256` | Type: int | A2, A3 | + +## Hierarchical cache + +| Argument | Defaults | Options | Server supported | +|-------------------------------------------------|-----------------|---------------------------------------------------------------------|:----------------:| +| `--enable-hierarchical-`
`cache` | `False` | bool flag
(set to enable) | A2, A3 | +| `--hicache-ratio` | `2.0` | Type: float | A2, A3 | +| `--hicache-size` | `0` | Type: int | A2, A3 | +| `--hicache-write-policy` | `write_through` | `write_back`,
`write_through`,
`write_through_selective` | A2, A3 | +| `--radix-eviction-policy` | `lru` | `lru`, `lfu` | A2, A3 | +| `--hicache-io-backend` | `kernel` | `kernel_ascend`,
`direct` | A2, A3 | +| `--hicache-mem-layout` | `layer_first` | `page_first_direct`,
`page_first_kv_split` | A2, A3 | +| `--hicache-storage-`
`backend` | `None` | `file` | A2, A3 | +| `--hicache-storage-`
`prefetch-policy` | `best_effort` | `best_effort`,
`wait_complete`,
`timeout` | Special for GPU | +| `--hicache-storage-`
`backend-extra-config` | `None` | Type: str | Special for GPU | + +## LMCache + +| Argument | Defaults | Options | Server supported | +|--------------------|----------|--------------------------------|:----------------:| +| `--enable-lmcache` | `False` | bool flag
(set to enable) | Special for GPU | + +## Offloading + +| Argument | Defaults | Options | Server supported | +|---------------------------|----------|-----------|:----------------:| +| `--cpu-offload-gb` | `0` | Type: int | A2, A3 | +| `--offload-group-size` | `-1` | Type: int | A2, A3 | +| `--offload-num-in-group` | `1` | Type: int | A2, A3 | +| `--offload-prefetch-step` | `1` | Type: int | A2, A3 | +| `--offload-mode` | `cpu` | Type: str | A2, A3 | + +## Args for multi-item scoring + +| Argument | Defaults | Options | Server supported | +|----------------------------------|----------|-----------|:----------------:| +| `--multi-item-scoring-delimiter` | `None` | Type: int | A2, A3 | + +## Optimization/debug options + +| Argument | Defaults | Options | Server supported | +|---------------------------------------------------------|----------|--------------------------------|:----------------:| +| `--disable-radix-cache` | `False` | bool flag
(set to enable) | A2, A3 | +| `--cuda-graph-max-bs` | `None` | Type: int | A2, A3 | +| `--cuda-graph-bs` | `None` | List[int] | A2, A3 | +| `--disable-cuda-graph` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-cuda-graph-`
`padding` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-profile-`
`cuda-graph` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-cudagraph-gc` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-nccl-nvls` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-symm-mem` | `False` | bool flag
(set to enable) | Special for GPU | +| `--disable-flashinfer-`
`cutlass-moe-fp4-allgather` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-outlines-`
`disk-cache` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-custom-`
`all-reduce` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-mscclpp` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-torch-`
`symm-mem` | `False` | bool flag
(set to enable) | Special for GPU | +| `--disable-overlap`
`-schedule` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-mixed-`
`chunk` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-dp-attention` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-dp-lm-head` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-two-`
`batch-overlap` | `False` | bool flag
(set to enable) | Planned | +| `--enable-single-`
`batch-overlap` | `False` | bool flag
(set to enable) | A2, A3 | +| `--tbo-token-`
`distribution-threshold` | `0.48` | Type: float | Planned | +| `--enable-torch-`
`compile` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-torch-`
`compile-debug-mode` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-piecewise-`
`cuda-graph` | `False` | bool flag
(set to enable) | A2, A3 | +| `--piecewise-cuda-`
`graph-tokens` | `None` | Type: JSON
list | A2, A3 | +| `--piecewise-cuda-`
`graph-compiler` | `eager` | ["eager", "inductor"] | A2, A3 | +| `--torch-compile-max-bs` | `32` | Type: int | A2, A3 | +| `--piecewise-cuda-`
`graph-max-tokens` | `4096` | Type: int | A2, A3 | +| `--torchao-config` | `` | Type: str | Special for GPU | +| `--enable-nan-detection` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-p2p-check` | `False` | bool flag
(set to enable) | Special for GPU | +| `--triton-attention-`
`reduce-in-fp32` | `False` | bool flag
(set to enable) | Special for GPU | +| `--triton-attention-`
`num-kv-splits` | `8` | Type: int | Special for GPU | +| `--triton-attention-`
`split-tile-size` | `None` | Type: int | Special for GPU | +| `--delete-ckpt-`
`after-loading` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-memory-saver` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-draft-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | A2, A3 | +| `--allow-auto-truncate` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-custom-`
`logit-processor` | `False` | bool flag
(set to enable) | A2, A3 | +| `--flashinfer-mla-`
`disable-ragged` | `False` | bool flag
(set to enable) | Special for GPU | +| `--disable-shared-`
`experts-fusion` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-chunked-`
`prefix-cache` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-fast-`
`image-processor` | `False` | bool flag
(set to enable) | A2, A3 | +| `--keep-mm-feature-`
`on-device` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-return-`
`hidden-states` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-return-`
`routed-experts` | `False` | bool flag
(set to enable) | A2, A3 | +| `--scheduler-recv-`
`interval` | `1` | Type: int | A2, A3 | +| `--numa-node` | `None` | List[int] | A2, A3 | +| `--rl-on-policy-target` | `None` | `fsdp` | Planned | +| `--enable-layerwise-`
`nvtx-marker` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-attn-tp-`
`input-scattered` | `False` | bool flag
(set to enable) | Experimental | +| `--enable-nsa-prefill-`
`context-parallel` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-fused-qk-`
`norm-rope` | `False` | bool flag
(set to enable) | Special for GPU | + +## Dynamic batch tokenizer + +| Argument | Defaults | Options | Server supported | +|--------------------------------------------------|----------|--------------------------------|:----------------:| +| `--enable-dynamic-`
`batch-tokenizer` | `False` | bool flag
(set to enable) | A2, A3 | +| `--dynamic-batch-`
`tokenizer-batch-size` | `32` | Type: int | A2, A3 | +| `--dynamic-batch-`
`tokenizer-batch-timeout` | `0.002` | Type: float | A2, A3 | + +## Debug tensor dumps + +| Argument | Defaults | Options | Server supported | +|--------------------------------------------|----------|-----------|:----------------:| +| `--debug-tensor-dump-`
`output-folder` | `None` | Type: str | A2, A3 | +| `--debug-tensor-dump-`
`layers` | `None` | List[int] | A2, A3 | +| `--debug-tensor-dump-`
`input-file` | `None` | Type: str | A2, A3 | + +## PD disaggregation + +| Argument | Defaults | Options | Server supported | +|---------------------------------------------------------|------------|---------------------------------------|:----------------:| +| `--disaggregation-mode` | `null` | `null`,
`prefill`,
`decode` | A2, A3 | +| `--disaggregation-transfer-backend` | `mooncake` | `ascend` | A2, A3 | +| `--disaggregation-bootstrap-port` | `8998` | Type: int | A2, A3 | +| `--disaggregation-decode-tp` | `None` | Type: int | A2, A3 | +| `--disaggregation-decode-dp` | `None` | Type: int | A2, A3 | +| `--disaggregation-ib-device` | `None` | Type: str | Special for GPU | +| `--disaggregation-decode-`
`enable-offload-kvcache` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disaggregation-decode-`
`enable-fake-auto` | `False` | bool flag
(set to enable) | A2, A3 | +| `--num-reserved-decode-tokens` | `512` | Type: int | A2, A3 | +| `--disaggregation-decode-`
`polling-interval` | `1` | Type: int | A2, A3 | + +## Encode prefill disaggregation + +| Argument | Defaults | Options | Server supported | +|------------------------------|--------------------|----------------------------------------------------------------|:----------------:| +| `--encoder-only` | `False` | bool flag
(set to enable) | A2, A3 | +| `--language-only` | `False` | bool flag
(set to enable) | A2, A3 | +| `--encoder-transfer-backend` | `zmq_to_scheduler` | `zmq_to_scheduler`,
`zmq_to_tokenizer`,
`mooncake` | A2, A3 | +| `--encoder-urls` | `[]` | List[str] | A2, A3 | + +## Custom weight loader + +| Argument | Defaults | Options | Server supported | +|-------------------------------------------------------------------------|----------|---------------------------------|:----------------:| +| `--custom-weight-loader` | `None` | List[str] | A2, A3 | +| `--weight-loader-disable-`
`mmap` | `False` | bool flag
(set to enable) | A2, A3 | +| `--remote-instance-weight-`
`loader-seed-instance-ip` | `None` | Type: str | A2, A3 | +| `--remote-instance-weight-`
`loader-seed-instance-service-port` | `None` | Type: int | A2, A3 | +| `--remote-instance-weight-`
`loader-send-weights-group-ports` | `None` | Type: JSON
list | A2, A3 | +| `--remote-instance-weight-`
`loader-backend` | `nccl` | `transfer_engine`,
`nccl` | A2, A3 | +| `--remote-instance-weight-`
`loader-start-seed-via-transfer-engine` | `False` | bool flag
(set to enable) | Special for GPU | + +## For PD-Multiplexing + +| Argument | Defaults | Options | Server supported | +|-----------------------|----------|--------------------------------|:----------------:| +| `--enable-pdmux` | `False` | bool flag
(set to enable) | Special for GPU | +| `--pdmux-config-path` | `None` | Type: str | Special for GPU | +| `--sm-group-num` | `8` | Type: int | Special for GPU | + +## For Multi-Modal + +| Argument | Defaults | Options | Server supported | +|-----------------------------------------------|----------|--------------------------------|:----------------:| +| `--mm-max-concurrent-calls` | 32 | Type: int | A2, A3 | +| `--mm-per-request-timeout` | 10.0 | Type: float | A2, A3 | +| `--enable-broadcast-mm-`
`inputs-process` | `False` | bool flag
(set to enable) | A2, A3 | +| `--mm-process-config` | `None` | Type: JSON / Dict | A2, A3 | +| `--mm-enable-dp-encoder` | `False` | bool flag
(set to enable) | A2, A3 | +| `--limit-mm-data-per-request` | `None` | Type: JSON / Dict | A2, A3 | + +## For checkpoint decryption + +| Argument | Defaults | Options | Server supported | +|---------------------------------|----------|--------------------------------|:----------------:| +| `--decrypted-config-file` | `None` | Type: str | A2, A3 | +| `--decrypted-draft-config-file` | `None` | Type: str | A2, A3 | +| `--enable-prefix-mm-cache` | `False` | bool flag
(set to enable) | A2, A3 | + +## For deterministic inference + +| Argument | Defaults | Options | Server supported | +|-------------------------------------------|----------|--------------------------------|:----------------:| +| `--enable-deterministic-`
`inference` | `False` | bool flag
(set to enable) | Planned | + +## For registering hooks + +| Argument | Defaults | Options | Server supported | +|-------------------|----------|-----------------|:----------------:| +| `--forward-hooks` | `None` | Type: JSON list | A2, A3 | + +## Configuration file support + +| Argument | Defaults | Options | Server supported | +|------------|----------|-----------|:----------------:| +| `--config` | `None` | Type: str | A2, A3 | + +## Other Params + +The following parameters are not supported because the third-party components that depend on are not compatible with the +NPU, like Ktransformer, checkpoint-engine etc. + +| Argument | Defaults | Options | +|-------------------------------------------------------------------|-----------|---------------------------| +| `--checkpoint-engine-`
`wait-weights-`
`before-ready` | `False` | bool flag (set to enable) | +| `--kt-weight-path` | `None` | Type: str | +| `--kt-method` | `AMXINT4` | Type: str | +| `--kt-cpuinfer` | `None` | Type: int | +| `--kt-threadpool-count` | 2 | Type: int | +| `--kt-num-gpu-experts` | `None` | Type: int | +| `--kt-max-deferred-`
`experts-per-token` | `None` | Type: int | + +The following parameters have some functional deficiencies on community + +| Argument | Defaults | Options | +|---------------------------------------|----------|--------------------------------| +| `--enable-double-sparsity` | `False` | bool flag
(set to enable) | +| `--ds-channel-config-path` | `None` | Type: str | +| `--ds-heavy-channel-num` | `32` | Type: int | +| `--ds-heavy-token-num` | `256` | Type: int | +| `--ds-heavy-channel-type` | `qk` | Type: str | +| `--ds-sparse-decode-`
`threshold` | `4096` | Type: int | +| `--tool-server` | `None` | Type: str | diff --git a/docs/platforms/ascend_npu_support_models.md b/docs/platforms/ascend_npu_support_models.md index 4a455d0c6..2e570b9ae 100644 --- a/docs/platforms/ascend_npu_support_models.md +++ b/docs/platforms/ascend_npu_support_models.md @@ -49,6 +49,11 @@ You are welcome to enable various models based on your business requirements. | OpenBMB/MiniCPM3-4B | MiniCPM (v3, 4B) | **√** | **√** | | Kimi/Kimi-K2-Thinking | Kimi | **√** | **√** | | openai/gpt-oss-120b | GPTOSS | **√** | **√** | +| allenai/OLMo-2-1124-7B-Instruct | OLMo | **√** | **√** | +| minimax/MiniMax-M2 | MiniMax-M2 | **√** | **√** | +| upstage/SOLAR-10.7B-Instruct-v1.0 | Solar | **√** | **√** | +| bigcode/starcoder2-7b | StarCoder2 | **√** | **√** | +| arcee-ai/Trinity-Mini | Trinity (Nano, Mini) | **√** | **√** | ## Multimodal Language Models @@ -75,6 +80,7 @@ You are welcome to enable various models based on your business requirements. | Kimi/Kimi-VL-A3B-Instruct | Kimi-VL (A3B) | **√** | **√** | | ZhipuAI/GLM-4.5V | GLM-4.5V (106B) | **√** | **√** | | LLM-Research/Llama-3.2-11B-Vision-Instruct | Llama 3.2 Vision (11B) | **√** | **√** | +| rednote-hilab/dots.ocr | DotsVLM-OCR | **√** | **√** | ## Embedding Models @@ -89,13 +95,13 @@ You are welcome to enable various models based on your business requirements. ## Reward Models -| Models | Model Family | A2 Supported | A3 Supported | -|---------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:| -| Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **√** | **√** | -| Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **√** | **√** | -| Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **√** | **√** | -| Howeee/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **√** | **√** | -| AI-ModelScope/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **√** | **√** | +| Models | Model Family | A2 Supported | A3 Supported | +|------------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:| +| Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **√** | **√** | +| Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **√** | **√** | +| Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **√** | **√** | +| Howeee/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **√** | **√** | +| AI-ModelScope/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **√** | **√** | ## Rerank Models