diff --git a/docs/platforms/ascend_npu_support_features.md b/docs/platforms/ascend_npu_support_features.md index 0dec8a312..b7cc2e87a 100644 --- a/docs/platforms/ascend_npu_support_features.md +++ b/docs/platforms/ascend_npu_support_features.md @@ -16,7 +16,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--tokenizer-worker-num` | `1` | Type: int | A2, A3 | | `--skip-tokenizer-init` | `False` | bool flag (set to enable) | A2, A3 | | `--load-format` | `auto` | `auto`, `safetensors` | A2, A3 | -| `--model-loader-`
`extra-config` | {} | Type: str | A2, A3 | +| `--model-loader-`
`extra-config` | `{}` | Type: str | A2, A3 | | `--trust-remote-code` | `False` | bool flag (set to enable) | A2, A3 | | `--context-length` | `None` | Type: int | A2, A3 | | `--is-embedding` | `False` | bool flag (set to enable) | A2, A3 | @@ -71,6 +71,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--page-size` | `128` | Type: int | A2, A3 | | `--swa-full-tokens-ratio` | `0.8` | Type: float | A2, A3 | | `--disable-hybrid-swa-memory` | `False` | bool flag
(set to enable) | A2, A3 | +| `--radix-eviction-policy` | `lru` | `lru`,
`lfu` | A2, A3 | | `--abort-on-priority-`
`when-disabled` | `False` | bool flag
(set to enable) | A2, A3 | | `--enable-dynamic-chunking` | `False` | bool flag
(set to enable) | A2, A3 | @@ -104,7 +105,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--log-level-http` | `None` | Type: str | A2, A3 | | `--log-requests` | `False` | bool flag
(set to enable) | A2, A3 | | `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | A2, A3 | -| `--log-requests-format` | text | text, json | A2, A3 | +| `--log-requests-format` | `text` | `text`, `json` | A2, A3 | | `--crash-dump-folder` | `None` | Type: str | A2, A3 | | `--enable-metrics` | `False` | bool flag
(set to enable) | A2, A3 | | `--enable-metrics-for-`
`all-schedulers` | `False` | bool flag
(set to enable) | A2, A3 | @@ -122,6 +123,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--kv-events-config` | `None` | Type: str | Special for GPU | | `--enable-trace` | `False` | bool flag
(set to enable) | A2, A3 | | `--oltp-traces-endpoint` | `localhost:4317` | Type: str | A2, A3 | +| `--log-requests-target` | `None` | Type: str | A2, A3 | +| `--uvicorn-access-log-exclude-prefixes` | `DEFAULT_UVICORN_ACCESS_LOG_EXCLUDE_PREFIXES` | Type: str | A2, A3 | ## RequestMetricsExporter configuration @@ -142,7 +145,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--completion-template` | `None` | Type: str | A2, A3 | | `--enable-cache-report` | `False` | bool flag
(set to enable) | A2, A3 | | `--reasoning-parser` | `None` | `deepseek-r1` | A2, A3 | -| `--tool-call-parser` | `None` | `llama`,`pythonic` | A2, A3 | +| `--tool-call-parser` | `None` | `deepseekv3`
`deepseekv31`
`glm`
`glm45`
`glm47`
`gpt-oss`
`kimi_k2`
`llama3`
`mistral`
`pythonic`
`qwen`
`qwen25`
`qwen3_coder`
`step3`
`gigachat3` | A2, A3 | | `--sampling-defaults` | `model` | `openai`, `model` | A2, A3 | ## Data parallelism @@ -150,7 +153,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | Argument | Defaults | Options | Server supported | |----------------------------------------|---------------|-----------------------------------------------------------|:----------------:| | `--data-parallel-size`
`--dp-size` | `1` | Type: int | A2, A3 | -| `--load-balance-method` | `round_robin` | `round_robin`,
`total_requests`,
`total_tokens` | A2, A3 | +| `--load-balance-method` | `auto` | `auto`,
`round_robin`,
`follow_bootstrap_room`,
`total_requests`,
`total_tokens` | A2, A3 | | `--prefill-round-robin-balance` | `False` | bool flag
(set to enable) | A2, A3 | ## Multi-node distributed serving @@ -179,7 +182,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--max-loras-per-batch` | `8` | Type: int | A2, A3 | | `--max-loaded-loras` | `None` | Type: int | A2, A3 | | `--lora-eviction-policy` | `lru` | `lru`,
`fifo` | A2, A3 | -| `--lora-backend` | `triton` | `triton` | A2, A3 | +| `--lora-backend` | `csgmv` | `triton`,
`csgmv`,
`ascend`,
`torch_native` | A2, A3 | | `--max-lora-chunk-size` | `16` | `16`, `32`,
`64`, `128` | Special for GPU | ## Kernel Backends (Attention, Sampling, Grammar, GEMM) @@ -259,9 +262,9 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | Argument | Defaults | Options | Server supported | |------------------------------|-----------|-----------------------------------------------|:----------------:| | `--max-mamba-cache-size` | `None` | Type: int | A2, A3 | -| `--mamba-ssm-dtype` | `float32` | `float32`,
`bfloat16` | A2, A3 | -| `--mamba-full-memory-ratio` | `0.2` | Type: float | A2, A3 | -| `--mamba-scheduler-strategy` | `auto` | `auto`,
`no_buffer`,
`extra_buffer` | A2, A3 | +| `--mamba-ssm-dtype` | `float32` | `float32`,
`bfloat16`,
`float16` | A2, A3 | +| `--mamba-full-memory-ratio` | `0.9` | Type: float | A2, A3 | +| `--mamba-scheduler-strategy` | `auto` | Only `auto`, `no_buffer` supported | A2, A3 | | `--mamba-track-interval` | `256` | Type: int | A2, A3 | ## Hierarchical cache @@ -271,8 +274,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--enable-hierarchical-`
`cache` | `False` | bool flag
(set to enable) | A2, A3 | | `--hicache-ratio` | `2.0` | Type: float | A2, A3 | | `--hicache-size` | `0` | Type: int | A2, A3 | -| `--hicache-write-policy` | `write_through` | `write_back`,
`write_through`,
`write_through_selective` | A2, A3 | -| `--radix-eviction-policy` | `lru` | `lru`, `lfu` | A2, A3 | +| `--hicache-write-policy` | `write_through` | Currently only `write_back` supported | A2, A3 | | `--hicache-io-backend` | `kernel` | `kernel_ascend`,
`direct` | A2, A3 | | `--hicache-mem-layout` | `layer_first` | `page_first_direct`,
`page_first_kv_split` | A2, A3 | | `--hicache-storage-`
`backend` | `None` | `file` | A2, A3 | @@ -290,10 +292,10 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | Argument | Defaults | Options | Server supported | |---------------------------|----------|-----------|:----------------:| | `--cpu-offload-gb` | `0` | Type: int | A2, A3 | -| `--offload-group-size` | `-1` | Type: int | A2, A3 | -| `--offload-num-in-group` | `1` | Type: int | A2, A3 | -| `--offload-prefetch-step` | `1` | Type: int | A2, A3 | -| `--offload-mode` | `cpu` | Type: str | A2, A3 | +| `--offload-group-size` | `-1` | Type: int | Planned | +| `--offload-num-in-group` | `1` | Type: int | Planned | +| `--offload-prefetch-step` | `1` | Type: int | Planned | +| `--offload-mode` | `cpu` | Type: str | Planned | ## Args for multi-item scoring @@ -316,9 +318,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--enable-symm-mem` | `False` | bool flag
(set to enable) | Special for GPU | | `--disable-flashinfer-`
`cutlass-moe-fp4-allgather` | `False` | bool flag
(set to enable) | Special for GPU | | `--enable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | A2, A3 | -| `--disable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | A2, A3 | -| `--disable-outlines-`
`disk-cache` | `False` | bool flag
(set to enable) | A2, A3 | -| `--disable-custom-`
`all-reduce` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-tokenizer-`
`batch-decode` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-custom-`
`all-reduce` | `False` | bool flag
(set to enable) | Special for GPU | | `--enable-mscclpp` | `False` | bool flag
(set to enable) | Special for GPU | | `--enable-torch-`
`symm-mem` | `False` | bool flag
(set to enable) | Special for GPU | | `--disable-overlap`
`-schedule` | `False` | bool flag
(set to enable) | A2, A3 | @@ -334,7 +335,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--piecewise-cuda-`
`graph-tokens` | `None` | Type: JSON
list | A2, A3 | | `--piecewise-cuda-`
`graph-compiler` | `eager` | ["eager", "inductor"] | A2, A3 | | `--torch-compile-max-bs` | `32` | Type: int | A2, A3 | -| `--piecewise-cuda-`
`graph-max-tokens` | `4096` | Type: int | A2, A3 | +| `--piecewise-cuda-`
`graph-max-tokens` | `None` | Type: int | A2, A3 | | `--torchao-config` | `` | Type: str | Special for GPU | | `--enable-nan-detection` | `False` | bool flag
(set to enable) | A2, A3 | | `--enable-p2p-check` | `False` | bool flag
(set to enable) | Special for GPU | @@ -356,6 +357,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--enable-return-`
`routed-experts` | `False` | bool flag
(set to enable) | A2, A3 | | `--scheduler-recv-`
`interval` | `1` | Type: int | A2, A3 | | `--numa-node` | `None` | List[int] | A2, A3 | +| `--enable-deterministic-`
`inference` | `False` | bool flag
(set to enable) | Planned | | `--rl-on-policy-target` | `None` | `fsdp` | Planned | | `--enable-layerwise-`
`nvtx-marker` | `False` | bool flag
(set to enable) | Special for GPU | | `--enable-attn-tp-`
`input-scattered` | `False` | bool flag
(set to enable) | Experimental | @@ -385,8 +387,6 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--disaggregation-mode` | `null` | `null`,
`prefill`,
`decode` | A2, A3 | | `--disaggregation-transfer-backend` | `mooncake` | `ascend` | A2, A3 | | `--disaggregation-bootstrap-port` | `8998` | Type: int | A2, A3 | -| `--disaggregation-decode-tp` | `None` | Type: int | A2, A3 | -| `--disaggregation-decode-dp` | `None` | Type: int | A2, A3 | | `--disaggregation-ib-device` | `None` | Type: str | Special for GPU | | `--disaggregation-decode-`
`enable-offload-kvcache` | `False` | bool flag
(set to enable) | A2, A3 | | `--disaggregation-decode-`
`enable-fake-auto` | `False` | bool flag
(set to enable) | A2, A3 | @@ -426,8 +426,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | Argument | Defaults | Options | Server supported | |-----------------------------------------------|----------|--------------------------------|:----------------:| -| `--mm-max-concurrent-calls` | 32 | Type: int | A2, A3 | -| `--mm-per-request-timeout` | 10.0 | Type: float | A2, A3 | +| `--mm-max-concurrent-calls` | `32` | Type: int | A2, A3 | +| `--mm-per-request-timeout` | `10.0` | Type: float | A2, A3 | | `--enable-broadcast-mm-`
`inputs-process` | `False` | bool flag
(set to enable) | A2, A3 | | `--mm-process-config` | `None` | Type: JSON / Dict | A2, A3 | | `--mm-enable-dp-encoder` | `False` | bool flag
(set to enable) | A2, A3 | @@ -441,13 +441,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--decrypted-draft-config-file` | `None` | Type: str | A2, A3 | | `--enable-prefix-mm-cache` | `False` | bool flag
(set to enable) | A2, A3 | -## For deterministic inference - -| Argument | Defaults | Options | Server supported | -|-------------------------------------------|----------|--------------------------------|:----------------:| -| `--enable-deterministic-`
`inference` | `False` | bool flag
(set to enable) | Planned | - -## For registering hooks +## Forward hooks | Argument | Defaults | Options | Server supported | |-------------------|----------|-----------------|:----------------:| diff --git a/docs/platforms/ascend_npu_support_models.md b/docs/platforms/ascend_npu_support_models.md index a30b6d21a..674436537 100644 --- a/docs/platforms/ascend_npu_support_models.md +++ b/docs/platforms/ascend_npu_support_models.md @@ -9,7 +9,7 @@ You are welcome to enable various models based on your business requirements. | Models | Model Family | A2 Supported | A3 Supported | |--------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:| | DeepSeek V3/V3.1 | DeepSeek | **√** | **√** | -| DeepSeek-V3.2-Exp-W8A8 | DeepSeek | **√** | **√** | +| DeepSeek-V3.2-W8A8 | DeepSeek | **√** | **√** | | DeepSeek-R1-0528-W8A8 | DeepSeek | **√** | **√** | | DeepSeek-V2-Lite-W8A8 | DeepSeek | **√** | **√** | | Qwen/Qwen3-30B-A3B-Instruct-2507 | Qwen | **√** | **√** |