update docs of Ascend plateform (#17358)
This commit is contained in:
amote-i
@@ -0,0 +1,152 @@
|
||||
# Contribution Guide
|
||||
|
||||
Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.
|
||||
|
||||
## Install SGLang from Source
|
||||
|
||||
### Prepare Environment
|
||||
|
||||
Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../platforms/ascend_npu.md) to install the necessary dependencies. we recommend [using docker](../platforms/ascend_npu.md#method-2-using-docker-image) to build the environment.
|
||||
|
||||
### Fork and clone the repository
|
||||
|
||||
**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/<your_user_name>/sglang.git
|
||||
# if you are using docker, the environment is already set up.
|
||||
cd sglang
|
||||
export PYTHONPATH=$PWD/python:$PYTHONPATH
|
||||
```
|
||||
|
||||
## Format code with pre-commit
|
||||
|
||||
We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:
|
||||
|
||||
```bash
|
||||
pip3 install pre-commit
|
||||
pre-commit install
|
||||
pre-commit run --all-files
|
||||
```
|
||||
|
||||
- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
|
||||
- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
|
||||
|
||||
## Run and add unit tests
|
||||
|
||||
If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression.
|
||||
SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework.
|
||||
For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
|
||||
|
||||
## Write documentations
|
||||
|
||||
We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase.
|
||||
For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md).
|
||||
|
||||
## Test the accuracy
|
||||
If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K.
|
||||
|
||||
```
|
||||
# Launch a server
|
||||
python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct
|
||||
|
||||
# Evaluate
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
|
||||
This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
|
||||
Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.
|
||||
|
||||
GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
|
||||
You can find additional accuracy eval examples in:
|
||||
- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_eval_accuracy_large.py)
|
||||
- [test_moe_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_moe_eval_accuracy_large.py)
|
||||
|
||||
## Benchmark the speed
|
||||
Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling.md).
|
||||
|
||||
## Requesting a review for merge
|
||||
You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
|
||||
You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
|
||||
Then your PR can be merged.
|
||||
|
||||
## How to Trigger CI Tests
|
||||
|
||||
We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
|
||||
Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)
|
||||
|
||||
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
|
||||
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI.
|
||||
- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit.
|
||||
- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`.
|
||||
- `/rerun-stage <stage-name>`: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete.
|
||||
|
||||
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
|
||||
|
||||
To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).
|
||||
|
||||
Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`.
|
||||
|
||||
If you don’t have permission, please ask maintainers to trigger CI for you.
|
||||
|
||||
### CI rate limits
|
||||
|
||||
Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
|
||||
|
||||
We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.
|
||||
|
||||
Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:
|
||||
|
||||
```yaml
|
||||
cool-down-minutes:
|
||||
description: "Default cooldown period in minutes; 0 disables rate limiting"
|
||||
type: number
|
||||
default: 120
|
||||
```
|
||||
|
||||
Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.
|
||||
|
||||
|
||||
## Code style guidance
|
||||
- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
|
||||
- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
|
||||
- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
|
||||
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible.
|
||||
- Make functions as pure as possible. Avoid in-place modification of arguments.
|
||||
- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`)
|
||||
- Keep tests run fast.
|
||||
- If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
|
||||
- If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
|
||||
- Reuse server launches in your unit tests to make tests run faster.
|
||||
- When supporting new hardware or features, follow these guidelines:
|
||||
- Do not drastically change existing code.
|
||||
- Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`).
|
||||
- If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.
|
||||
|
||||
## How to update sgl-kernel
|
||||
Since sglang and sgl-kernel are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR).
|
||||
To add a new kernel or modify an existing one in the sgl-kernel package, you must use multiple PRs.
|
||||
|
||||
Follow these steps:
|
||||
|
||||
1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)).
|
||||
2. Bump the version of sgl-kernel (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)).
|
||||
- Once merged, this will trigger an automatic release of the sgl-kernel wheel to PyPI.
|
||||
- If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week.
|
||||
3. Apply the changes:
|
||||
- Update the sgl-kernel version in `sglang/python/pyproject.toml` to use the modified kernels.
|
||||
- Update the related caller code in the sglang to use the new kernel.
|
||||
|
||||
## How to update sgl-kernel-npu
|
||||
|
||||
Sgl-kernel-npu is the kernel package for Ascend NPU and is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. if you want to add a new kernel and want to use it in sglang, please follow the steps in [Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md).
|
||||
|
||||
## Tips for newcomers
|
||||
|
||||
If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out this [code walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
|
||||
|
||||
If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).
|
||||
|
||||
Thank you for your interest in SGLang. Happy coding!
|
||||
@@ -6,10 +6,10 @@ You can install SGLang using any of the methods below. Please go through `System
|
||||
## Component Version Mapping For SGLang
|
||||
| Component | Version | Obtain Way |
|
||||
|-------------------|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| HDK | 25.2.1 | [link](https://support.huawei.com/carrier/productNewOffering?col=product&path=PBI1-262732867/PBI1-262735886/PBI1-262735910/PBI1-261410188/PBI1-252764743&pVR=PBI1-263550357&pC=PBI1-264360782&pSPC=PBI1-266220744&resTab=SW) |
|
||||
| HDK | 25.3.RC1 | [link](https://hiascend.com/hardware/firmware-drivers/commercial?product=7&model=33) |
|
||||
| CANN | 8.3.rc2 | [Obtain Images](#obtain-cann-image) |
|
||||
| Pytorch Adapter | 7.3.0 | [link](https://gitcode.com/Ascend/pytorch/releases) |
|
||||
| MemFabric | 0.1.0 | [link](https://gitcode.com/Ascend/memfabric_hybrid/releases) |
|
||||
| MemFabric | 1.0.3 | `pip install memfabric-hybrid==1.0.3` |
|
||||
| Triton | 3.2.0.dev2025112116 | [link](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/triton_ascend-3.2.0.dev2025112116-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl) |
|
||||
| Bisheng | 20251121 | [link](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/Ascend-BiSheng-toolkit_aarch64_20251121.run) |
|
||||
| SGLang NPU Kernel | NA | [link](https://github.com/sgl-project/sgl-kernel-npu/releases) |
|
||||
@@ -46,32 +46,17 @@ Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kern
|
||||
If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
|
||||
|
||||
```shell
|
||||
pip install memfabric-hybrid==1.0.0
|
||||
pip install memfabric-hybrid==1.0.3
|
||||
```
|
||||
|
||||
#### Pytorch and Pytorch Framework Adaptor on Ascend
|
||||
|
||||
At the moment NPUGraph optimizations are supported only in `torch_npu==2.6.0.post3` that requires 'torch==2.6.0'.
|
||||
|
||||
_TODO: NPUGraph optimizations will be supported in future releases of 'torch_npu' 2.7.1, 2.8.0 and 2.9.0_
|
||||
|
||||
```shell
|
||||
PYTORCH_VERSION=2.6.0
|
||||
TORCHVISION_VERSION=0.21.0
|
||||
TORCH_NPU_VERSION=2.6.0.post3
|
||||
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install torch_npu==$TORCH_NPU_VERSION
|
||||
```
|
||||
|
||||
While there is no released versions of 'torch_npu' for 'torch==2.7.1' and 'torch==2.8.0' we provide custom builds of 'torch_npu'. PLATFORM can be 'aarch64' or 'x86_64'
|
||||
|
||||
```shell
|
||||
PLATFORM="aarch64"
|
||||
PYTORCH_VERSION=2.8.0
|
||||
TORCHVISION_VERSION=0.23.0
|
||||
TORCH_NPU_VERSION=2.8.0
|
||||
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
|
||||
wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/torch_npu/torch_npu-${PYTORCH_VERSION}.post2.dev20251120-cp311-cp311-manylinux_2_28_${PLATFORM}.whl
|
||||
pip install torch_npu-${PYTORCH_VERSION}.post2.dev20251120-cp311-cp311-manylinux_2_28_${PLATFORM}.whl
|
||||
pip install torch_npu==$TORCH_NPU_VERSION
|
||||
```
|
||||
|
||||
If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md)
|
||||
@@ -86,7 +71,7 @@ BISHENG_URL="https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton
|
||||
wget -O "${BISHENG_NAME}" "${BISHENG_URL}" && chmod a+x "${BISHENG_NAME}" && "./${BISHENG_NAME}" --install && rm "${BISHENG_NAME}"
|
||||
```
|
||||
```shell
|
||||
pip install triton-ascend==3.2.0rc4
|
||||
pip install -i https://test.pypi.org/simple/ "triton-ascend<3.2.0rc" --pre --no-cache-dir
|
||||
```
|
||||
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md)
|
||||
|
||||
@@ -102,18 +87,18 @@ Additional package with custom operations. DEVICE_TYPE can be "a3" for Atlas A3
|
||||
|
||||
```shell
|
||||
DEVICE_TYPE="a3"
|
||||
wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run
|
||||
chmod a+x ./CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run
|
||||
./CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run --quiet --install-path=/usr/local/Ascend/ascend-toolkit/latest/opp
|
||||
wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/custom_ops-1.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl
|
||||
pip install ./custom_ops-1.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl
|
||||
wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/CANN-custom_ops-8.3.0.1-$DEVICE_TYPE-linux.aarch64.run
|
||||
chmod a+x ./CANN-custom_ops-8.3.0.1-$DEVICE_TYPE-linux.aarch64.run
|
||||
./CANN-custom_ops-8.3.0.1-$DEVICE_TYPE-linux.aarch64.run --quiet --install-path=/usr/local/Ascend/ascend-toolkit/latest/opp
|
||||
wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/custom_ops-2.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl
|
||||
pip install ./custom_ops-2.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl
|
||||
```
|
||||
|
||||
#### Installing SGLang from source
|
||||
|
||||
```shell
|
||||
# Use the last release branch
|
||||
git clone -b v0.5.6.post2 https://github.com/sgl-project/sglang.git
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
mv python/pyproject_other.toml python/pyproject.toml
|
||||
pip install -e python[srt_npu]
|
||||
@@ -137,6 +122,7 @@ git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang/docker
|
||||
|
||||
# Build the docker image
|
||||
# If there are network errors, please modify the Dockerfile to use offline dependencies or use a proxy
|
||||
docker build -t <image_name> -f npu.Dockerfile .
|
||||
```
|
||||
|
||||
@@ -224,6 +210,8 @@ python3 -m sglang.launch_server \
|
||||
--device npu \
|
||||
--base-gpu-id 0 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
2. Launch Decode Server
|
||||
|
||||
@@ -30,6 +30,7 @@ python3 -m sglang.launch_server \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--watchdog-timeout 9000 \
|
||||
--cuda-graph-bs 8 16 24 28 32 \
|
||||
--mem-fraction-static 0.68 \
|
||||
@@ -87,6 +88,8 @@ python -m sglang.launch_server \
|
||||
--mem-fraction-static 0.6 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--load-balance-method round_robin \
|
||||
--max-running-requests 8 \
|
||||
--context-length 8192 \
|
||||
--disable-radix-cache \
|
||||
@@ -142,6 +145,8 @@ python -m sglang.launch_server \
|
||||
--max-running-requests 352 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--prefill-round-robin-balance \
|
||||
--moe-a2a-backend deepep \
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
@@ -195,7 +200,7 @@ export ENABLE_MOE_NZ=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
#Please list all host ips of Prefill instance
|
||||
P_HOST_IP=('xx,xx,xx,xx' 'xx,xx,xx,xx')
|
||||
P_HOST_IP=('xx.xx.xx.xx' 'xx.xx.xx.xx')
|
||||
|
||||
for i in "${!P_HOST_IP[@]}";
|
||||
do
|
||||
@@ -255,7 +260,7 @@ export SGLANG_USE_FIA_NZ=1
|
||||
export ENABLE_MOE_NZ=1
|
||||
|
||||
#please list all host ips of Prefill instance
|
||||
D_HOST_IP=('xx,xx,xx,xx' 'xx,xx,xx,xx')
|
||||
D_HOST_IP=('xx.xx.xx.xx' 'xx.xx.xx.xx')
|
||||
|
||||
for i in "${!D_HOST_IP[@]}";
|
||||
do
|
||||
|
||||
@@ -9,4 +9,5 @@ Ascend NPUs
|
||||
ascend_npu_support_features.md
|
||||
ascend_npu_deepseek_example.md
|
||||
ascend_npu_qwen3_examples.md
|
||||
ascend_contribution_guide.md
|
||||
ascend_npu_best_practice.md
|
||||
|
||||
@@ -70,10 +70,10 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--priority-scheduling-`<br/>`preemption-threshold` | `10` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--schedule-conservativeness` | `1.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--page-size` | `128` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--abort-on-priority-`<br/>`when-disabled` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--hybrid-kvcache-ratio` | `None` | Optional[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--swa-full-tokens-ratio` | `0.8` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-hybrid-swa-memory` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--abort-on-priority-`<br/>`when-disabled` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-dynamic-chunking` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Runtime options
|
||||
@@ -83,7 +83,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--device` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--pp-max-micro-batch-size` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--pp-max-micro-batch-size` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--pp-async-batch-depth` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--stream-interval` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--stream-output` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -91,7 +91,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--constrained-json-`<br/>`whitespace-pattern` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--constrained-json-`<br/>`disable-any-whitespace` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--watchdog-timeout` | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--soft-watchdog-timeout` | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--soft-watchdog-timeout` | `300` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--download-dir` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--dist-timeout` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--base-gpu-id` | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--gpu-id-step` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -106,21 +107,22 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--log-level-http` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--log-requests` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--log-requests-format` | text | text, json | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--crash-dump-folder` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--log-requests-format` | text | text, json | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--crash-dump-folder` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--show-time-cost` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--crash-on-nan` | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-metrics` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-metrics-for-`<br/>`all-schedulers` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tokenizer-metrics-`<br/>`custom-labels-header` | `x-custom-labels` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tokenizer-metrics-`<br/>`allowed-custom-labels` | `None` | List[str] | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--bucket-time-to-`<br/>`first-token` | `None` | List[float] | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--bucket-inter-token-`<br/>`latency` | `None` | List[float] | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--bucket-e2e-request-`<br/>`latency` | `None` | List[float] | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--collect-tokens-`<br/>`histogram` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--prompt-tokens-buckets` | `None` | List[str] | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--generation-tokens-buckets` | `None` | List[str] | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--gc-warning-threshold-secs` | `0.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--decode-log-interval` | `40` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-metrics` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-metrics-for-`<br/>`all-schedulers` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--tokenizer-metrics-`<br/>`custom-labels-header` | `x-custom-labels` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--tokenizer-metrics-`<br/>`allowed-custom-labels` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--bucket-time-to-`<br/>`first-token` | `None` | List[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--bucket-inter-token-`<br/>`latency` | `None` | List[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--bucket-e2e-request-`<br/>`latency` | `None` | List[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--collect-tokens-`<br/>`histogram` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--prompt-tokens-buckets` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--generation-tokens-buckets` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--gc-warning-threshold-secs` | `0.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--decode-log-interval` | `40` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-request-time-`<br/>`stats-logging` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--kv-events-config` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-trace` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -128,26 +130,26 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
|
||||
## RequestMetricsExporter configuration
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|---------------------------------------|----------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|---------------------------------------|----------|--------------------------------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--export-metrics-to-`<br/>`file` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--export-metrics-to-`<br/>`file-dir` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## API related
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------|-----------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--api-key` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--admin-api-key` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--served-model-name` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--weight-version` | `default` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--chat-template` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--completion-template` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-cache-report` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--reasoning-parser` | `None` | `deepseek-r1` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tool-call-parser` | `None` | `llama`,`pythonic` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--sampling-defaults` | `model` | `openai`, `model` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tool-server` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------|------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--api-key` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--served-model-name` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--weight-version` | `default` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--chat-template` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--completion-template` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--file-storage-path` | `sglang_storage` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-cache-report` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--reasoning-parser` | `None` | `deepseek-r1` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tool-call-parser` | `None` | `llama`,`pythonic` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--sampling-defaults` | `model` | `openai`, `model` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--tool-server` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Data parallelism
|
||||
|
||||
@@ -155,7 +157,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
|----------------------------------------|---------------|-----------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--data-parallel-size`<br/>`--dp-size` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--load-balance-method` | `round_robin` | `round_robin`,<br/> `total_requests`,<br/> `total_tokens` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--prefill-round-robin-balance` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--load-watch-interval` | `0.1` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--prefill-round-robin-balance` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Multi-node distributed serving
|
||||
|
||||
@@ -182,7 +185,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--lora-paths` | `None` | Type: List[str] /<br/> JSON objects | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--max-loras-per-batch` | `8` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--max-loaded-loras` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--lora-eviction-policy` | `lru` | `lru`,<br/> `fifo` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--lora-eviction-policy` | `lru` | `lru`,<br/> `fifo` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--lora-backend` | `triton` | `triton` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--max-lora-chunk-size` | `16` | `16`, `32`,<br/> `64`, `128` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
@@ -212,9 +215,9 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--speculative-num-steps` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-eagle-topk` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-num-draft-tokens` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-accept-`<br/>`threshold-single` | `1.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--speculative-accept-`<br/>`threshold-acc` | `1.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--speculative-token-map` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-accept-`<br/>`threshold-single` | `1.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-accept-`<br/>`threshold-acc` | `1.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-token-map` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--speculative-attention-`<br/>`mode` | `prefill` | `prefill`,<br/> `decode` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-moe-runner-`<br/>`backend` | `None` | `auto` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--speculative-moe-a2a-`<br/>`backend` | `None` | `ascend_fuseep` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -233,30 +236,37 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--speculative-ngram-`<br/>`branch-length` | `18` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--speculative-ngram-`<br/>`capacity` | `10000000` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Multi-layer Eagle speculative decoding
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|---------------------------------------------------|----------|-------------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--enable-multi-layer-eagle` | `False` | Bool flag <br/>(set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Expert parallelism
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------------------------------------|-----------|---------------------------------------------|:-----------------------------------------:|:----------------------------------------:|
|
||||
| `--expert-parallel-size`<br/>`--ep-size`<br/>`--ep` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--moe-a2a-backend` | `none` | `none`,<br/> `deepep`,<br/> `ascend_fuseep` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--moe-runner-backend` | `auto` | `auto`, `triton` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--flashinfer-mxfp4-`<br/>`moe-precision` | `default` | `default`,<br/> `bf16` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-flashinfer-`<br/>`allreduce-fusion` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--deepep-mode` | `auto` | `normal`, <br/>`low_latency`,<br/> `auto` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--deepep-config` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--ep-num-redundant-experts` | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--ep-dispatch-algorithm` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--init-expert-location` | `trivial` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-eplb` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--eplb-algorithm` | `auto` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--eplb-rebalance-layers-`<br/>`per-chunk` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--eplb-min-rebalancing-`<br/>`utilization-threshold` | `1.0` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--expert-distribution-`<br/>`recorder-mode` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--expert-distribution-`<br/>`recorder-buffer-size` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-expert-distribution-`<br/>`metrics` | `False` | bool flag <br/>(set to enable) | ***<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--moe-dense-tp-size` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--elastic-ep-backend` | `None` | `none`, `mooncake` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--mooncake-ib-device` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------------------------------------|-----------|---------------------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--expert-parallel-size`<br/>`--ep-size`<br/>`--ep` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--moe-a2a-backend` | `none` | `none`,<br/> `deepep`,<br/> `ascend_fuseep` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--moe-runner-backend` | `auto` | `auto`, `triton` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--flashinfer-mxfp4-`<br/>`moe-precision` | `default` | `default`,<br/> `bf16` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-flashinfer-`<br/>`allreduce-fusion` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--deepep-mode` | `auto` | `normal`, <br/>`low_latency`,<br/> `auto` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--deepep-config` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--ep-num-redundant-experts` | `0` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--ep-dispatch-algorithm` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--init-expert-location` | `trivial` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-eplb` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--eplb-algorithm` | `auto` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--eplb-rebalance-num-iterations` | `1000` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--eplb-rebalance-layers-`<br/>`per-chunk` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--eplb-min-rebalancing-`<br/>`utilization-threshold` | `1.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--expert-distribution-`<br/>`recorder-mode` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--expert-distribution-`<br/>`recorder-buffer-size` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-expert-distribution-`<br/>`metrics` | `False` | bool flag <br/>(set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--moe-dense-tp-size` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--elastic-ep-backend` | `None` | `none`, `mooncake` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--mooncake-ib-device` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Mamba Cache
|
||||
|
||||
@@ -300,6 +310,12 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--kt-num-gpu-experts` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--kt-max-deferred-`<br/>`experts-per-token` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Diffusion LLM
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|----------------------------------------------|-----------|-----------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--dllm-algorithm` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--dllm-algorithm-config` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Double Sparsity
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
@@ -324,8 +340,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
## Args for multi-item scoring
|
||||
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|----------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
|
||||
| `--multi-item-scoring-delimiter` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|----------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
|
||||
| `--multi-item-scoring-delimiter` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Optimization/debug options
|
||||
|
||||
@@ -337,13 +353,13 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--disable-cuda-graph` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disable-cuda-graph-`<br/>`padding` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-profile-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-cudagraph-gc` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-cudagraph-gc` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-nccl-nvls` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-symm-mem` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-flashinfer-`<br/>`cutlass-moe-fp4-allgather` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disable-outlines-`<br/>`disk-cache` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-outlines-`<br/>`disk-cache` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-custom-`<br/>`all-reduce` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-mscclpp` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-torch-`<br/>`symm-mem` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
@@ -352,8 +368,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--enable-dp-attention` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-dp-lm-head` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-two-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-single-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--tbo-token-`<br/>`distribution-threshold` | `0.48` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-single-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--tbo-token-`<br/>`distribution-threshold` | `0.48` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-torch-`<br/>`compile` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-torch-`<br/>`compile-debug-mode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-piecewise-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
@@ -362,26 +378,26 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--torch-compile-max-bs` | `32` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--piecewise-cuda-`<br/>`graph-max-tokens` | `4096` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--torchao-config` | `` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-nan-detection` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-nan-detection` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-p2p-check` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--triton-attention-`<br/>`reduce-in-fp32` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--triton-attention-`<br/>`num-kv-splits` | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--triton-attention-`<br/>`split-tile-size` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--num-continuous-`<br/>`decode-steps` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-memory-saver` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-draft-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-shared-`<br/>`experts-fusion` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-chunked-`<br/>`prefix-cache` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--disable-shared-`<br/>`experts-fusion` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disable-chunked-`<br/>`prefix-cache` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--enable-return-`<br/>`routed-experts` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--scheduler-recv-`<br/>`interval` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-return-`<br/>`routed-experts` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--scheduler-recv-`<br/>`interval` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--numa-node` | `None` | List[int] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--rl-on-policy-target` | `None` | `fsdp` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--enable-layerwise-`<br/>`nvtx-marker` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
@@ -434,7 +450,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| Argument | Defaults | Options | A2 | A3 |
|
||||
|-------------------------------------------------------------------------|----------|---------------------------------|:----------------------------------------:|------------------------------------------|
|
||||
| `--custom-weight-loader` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
| `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
@@ -22,6 +22,7 @@ You are welcome to enable various models based on your business requirements.
|
||||
| vllm-ascend/QWQ-32B-W8A8 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/Llama-3.1-8B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LLM-Research/llama-2-7b | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LLM-Research/Llama-3.2-1B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| mistralai/Mistral-7B-Instruct-v0.2 | Mistral | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| google/gemma-3-4b-it | Gemma | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -42,11 +43,12 @@ You are welcome to enable various models based on your business requirements.
|
||||
| inclusionAI/Ling-lite | Ling | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ibm-granite/granite-3.1-8b-instruct | Granite | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ibm-granite/granite-3.0-3b-a800m-instruct | Granite MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| databricks/dbrx-instruct | DBRX (Databricks) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/dbrx-instruct | DBRX (Databricks) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| baichuan-inc/Baichuan2-13B-Chat | Baichuan 2 (7B, 13B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| baidu/ERNIE-4.5-21B-A3B-PT | ERNIE-4.5 (4.5, 4.5MoE series) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM3-4B | MiniCPM (v3, 4B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openai/gpt-oss-120b | GPTOSS | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Kimi/Kimi-K2-Think | Kimi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openai/gpt-oss-120b | GPTOSS | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
|
||||
|
||||
## Multimodal Language Models
|
||||
|
||||
@@ -59,8 +61,10 @@ You are welcome to enable various models based on your business requirements.
|
||||
| Qwen/Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/Janus-Pro-1B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM-o-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| google/gemma-3-4b-it | Gemma 3 (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.1-24B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| microsoft/Phi-4-multimodal-instruct | Phi-4-multimodal-instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
@@ -90,7 +94,7 @@ You are welcome to enable various models based on your business requirements.
|
||||
| Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| jason9693/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Howeee/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Skywork/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Rerank Models
|
||||
|
||||
Reference in New Issue
Block a user