diff --git a/docs/platforms/ascend_contribution_guide.md b/docs/platforms/ascend_contribution_guide.md new file mode 100644 index 000000000..db3431260 --- /dev/null +++ b/docs/platforms/ascend_contribution_guide.md @@ -0,0 +1,152 @@ +# Contribution Guide + +Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process. + +## Install SGLang from Source + +### Prepare Environment + +Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../platforms/ascend_npu.md) to install the necessary dependencies. we recommend [using docker](../platforms/ascend_npu.md#method-2-using-docker-image) to build the environment. + +### Fork and clone the repository + +**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally. + +```bash +git clone https://github.com//sglang.git +# if you are using docker, the environment is already set up. +cd sglang +export PYTHONPATH=$PWD/python:$PYTHONPATH +``` + +## Format code with pre-commit + +We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run: + +```bash +pip3 install pre-commit +pre-commit install +pre-commit run --all-files +``` + +- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request. +- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch. + +## Run and add unit tests + +If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression. +SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework. +For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). + +## Write documentations + +We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase. +For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md). + +## Test the accuracy +If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K. + +``` +# Launch a server +python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct + +# Evaluate +python3 -m sglang.test.few_shot_gsm8k --num-questions 200 +``` + +Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test. +This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine. +Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test. + +GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests. +You can find additional accuracy eval examples in: +- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_eval_accuracy_large.py) +- [test_moe_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_moe_eval_accuracy_large.py) + +## Benchmark the speed +Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling.md). + +## Requesting a review for merge +You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md). +You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals. +Then your PR can be merged. + +## How to Trigger CI Tests + +We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests. +Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) + +For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands: + +- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI. +- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit. +- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`. +- `/rerun-stage `: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete. + +If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a πŸ‘ to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302). + +To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`). + +Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`. + +If you don’t have permission, please ask maintainers to trigger CI for you. + +### CI rate limits + +Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests. + +We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources. + +Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter: + +```yaml +cool-down-minutes: + description: "Default cooldown period in minutes; 0 disables rate limiting" + type: number + default: 120 +``` + +Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval. + + +## Code style guidance +- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function. +- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. +- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code. + - A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible. +- Make functions as pure as possible. Avoid in-place modification of arguments. +- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`) +- Keep tests run fast. + - If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`). + - If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps. + - Reuse server launches in your unit tests to make tests run faster. +- When supporting new hardware or features, follow these guidelines: + - Do not drastically change existing code. + - Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`). + - If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch. + +## How to update sgl-kernel +Since sglang and sgl-kernel are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR). +To add a new kernel or modify an existing one in the sgl-kernel package, you must use multiple PRs. + +Follow these steps: + +1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)). +2. Bump the version of sgl-kernel (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)). + - Once merged, this will trigger an automatic release of the sgl-kernel wheel to PyPI. + - If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week. +3. Apply the changes: + - Update the sgl-kernel version in `sglang/python/pyproject.toml` to use the modified kernels. + - Update the related caller code in the sglang to use the new kernel. + +## How to update sgl-kernel-npu + +Sgl-kernel-npu is the kernel package for Ascend NPU and is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. if you want to add a new kernel and want to use it in sglang, please follow the steps in [Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md). + +## Tips for newcomers + +If you want to contribute but don’t have a specific idea in mind, pick issues labeled [β€œgood first issue” or β€œhelp wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out this [code walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow. + +If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io). + +Thank you for your interest in SGLang. Happy coding! diff --git a/docs/platforms/ascend_npu.md b/docs/platforms/ascend_npu.md index 8f061bf27..3152f1218 100644 --- a/docs/platforms/ascend_npu.md +++ b/docs/platforms/ascend_npu.md @@ -6,10 +6,10 @@ You can install SGLang using any of the methods below. Please go through `System ## Component Version Mapping For SGLang | Component | Version | Obtain Way | |-------------------|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| HDK | 25.2.1 | [link](https://support.huawei.com/carrier/productNewOffering?col=product&path=PBI1-262732867/PBI1-262735886/PBI1-262735910/PBI1-261410188/PBI1-252764743&pVR=PBI1-263550357&pC=PBI1-264360782&pSPC=PBI1-266220744&resTab=SW) | +| HDK | 25.3.RC1 | [link](https://hiascend.com/hardware/firmware-drivers/commercial?product=7&model=33) | | CANN | 8.3.rc2 | [Obtain Images](#obtain-cann-image) | | Pytorch Adapter | 7.3.0 | [link](https://gitcode.com/Ascend/pytorch/releases) | -| MemFabric | 0.1.0 | [link](https://gitcode.com/Ascend/memfabric_hybrid/releases) | +| MemFabric | 1.0.3 | `pip install memfabric-hybrid==1.0.3` | | Triton | 3.2.0.dev2025112116 | [link](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/triton_ascend-3.2.0.dev2025112116-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl) | | Bisheng | 20251121 | [link](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/Ascend-BiSheng-toolkit_aarch64_20251121.run) | | SGLang NPU Kernel | NA | [link](https://github.com/sgl-project/sgl-kernel-npu/releases) | @@ -46,32 +46,17 @@ Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kern If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters. ```shell -pip install memfabric-hybrid==1.0.0 +pip install memfabric-hybrid==1.0.3 ``` #### Pytorch and Pytorch Framework Adaptor on Ascend -At the moment NPUGraph optimizations are supported only in `torch_npu==2.6.0.post3` that requires 'torch==2.6.0'. - -_TODO: NPUGraph optimizations will be supported in future releases of 'torch_npu' 2.7.1, 2.8.0 and 2.9.0_ - ```shell -PYTORCH_VERSION=2.6.0 -TORCHVISION_VERSION=0.21.0 -TORCH_NPU_VERSION=2.6.0.post3 -pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu -pip install torch_npu==$TORCH_NPU_VERSION -``` - -While there is no released versions of 'torch_npu' for 'torch==2.7.1' and 'torch==2.8.0' we provide custom builds of 'torch_npu'. PLATFORM can be 'aarch64' or 'x86_64' - -```shell -PLATFORM="aarch64" PYTORCH_VERSION=2.8.0 TORCHVISION_VERSION=0.23.0 +TORCH_NPU_VERSION=2.8.0 pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu -wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/torch_npu/torch_npu-${PYTORCH_VERSION}.post2.dev20251120-cp311-cp311-manylinux_2_28_${PLATFORM}.whl -pip install torch_npu-${PYTORCH_VERSION}.post2.dev20251120-cp311-cp311-manylinux_2_28_${PLATFORM}.whl +pip install torch_npu==$TORCH_NPU_VERSION ``` If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md) @@ -86,7 +71,7 @@ BISHENG_URL="https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton wget -O "${BISHENG_NAME}" "${BISHENG_URL}" && chmod a+x "${BISHENG_NAME}" && "./${BISHENG_NAME}" --install && rm "${BISHENG_NAME}" ``` ```shell -pip install triton-ascend==3.2.0rc4 +pip install -i https://test.pypi.org/simple/ "triton-ascend<3.2.0rc" --pre --no-cache-dir ``` For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md) @@ -102,18 +87,18 @@ Additional package with custom operations. DEVICE_TYPE can be "a3" for Atlas A3 ```shell DEVICE_TYPE="a3" -wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run -chmod a+x ./CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run -./CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run --quiet --install-path=/usr/local/Ascend/ascend-toolkit/latest/opp -wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/custom_ops-1.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl -pip install ./custom_ops-1.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl +wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/CANN-custom_ops-8.3.0.1-$DEVICE_TYPE-linux.aarch64.run +chmod a+x ./CANN-custom_ops-8.3.0.1-$DEVICE_TYPE-linux.aarch64.run +./CANN-custom_ops-8.3.0.1-$DEVICE_TYPE-linux.aarch64.run --quiet --install-path=/usr/local/Ascend/ascend-toolkit/latest/opp +wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/custom_ops-2.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl +pip install ./custom_ops-2.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl ``` #### Installing SGLang from source ```shell # Use the last release branch -git clone -b v0.5.6.post2 https://github.com/sgl-project/sglang.git +git clone https://github.com/sgl-project/sglang.git cd sglang mv python/pyproject_other.toml python/pyproject.toml pip install -e python[srt_npu] @@ -137,6 +122,7 @@ git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image +# If there are network errors, please modify the Dockerfile to use offline dependencies or use a proxy docker build -t -f npu.Dockerfile . ``` @@ -224,6 +210,8 @@ python3 -m sglang.launch_server \ --device npu \ --base-gpu-id 0 \ --tp-size 1 \ + --host 127.0.0.1 \ + --port 8000 ``` 2. Launch Decode Server diff --git a/docs/platforms/ascend_npu_deepseek_example.md b/docs/platforms/ascend_npu_deepseek_example.md index 612f0ed6f..a3beb31b2 100644 --- a/docs/platforms/ascend_npu_deepseek_example.md +++ b/docs/platforms/ascend_npu_deepseek_example.md @@ -30,6 +30,7 @@ python3 -m sglang.launch_server \ --trust-remote-code \ --attention-backend ascend \ --device npu \ + --quantization modelslim \ --watchdog-timeout 9000 \ --cuda-graph-bs 8 16 24 28 32 \ --mem-fraction-static 0.68 \ @@ -87,6 +88,8 @@ python -m sglang.launch_server \ --mem-fraction-static 0.6 \ --attention-backend ascend \ --device npu \ + --quantization modelslim \ + --load-balance-method round_robin \ --max-running-requests 8 \ --context-length 8192 \ --disable-radix-cache \ @@ -142,6 +145,8 @@ python -m sglang.launch_server \ --max-running-requests 352 \ --attention-backend ascend \ --device npu \ + --quantization modelslim \ + --prefill-round-robin-balance \ --moe-a2a-backend deepep \ --enable-dp-attention \ --deepep-mode low_latency \ @@ -195,7 +200,7 @@ export ENABLE_MOE_NZ=1 export TASK_QUEUE_ENABLE=2 #Please list all host ips of Prefill instance -P_HOST_IP=('xx,xx,xx,xx' 'xx,xx,xx,xx') +P_HOST_IP=('xx.xx.xx.xx' 'xx.xx.xx.xx') for i in "${!P_HOST_IP[@]}"; do @@ -255,7 +260,7 @@ export SGLANG_USE_FIA_NZ=1 export ENABLE_MOE_NZ=1 #please list all host ips of Prefill instance -D_HOST_IP=('xx,xx,xx,xx' 'xx,xx,xx,xx') +D_HOST_IP=('xx.xx.xx.xx' 'xx.xx.xx.xx') for i in "${!D_HOST_IP[@]}"; do diff --git a/docs/platforms/ascend_npu_support.rst b/docs/platforms/ascend_npu_support.rst index e786a1ef7..26306c7c1 100644 --- a/docs/platforms/ascend_npu_support.rst +++ b/docs/platforms/ascend_npu_support.rst @@ -9,4 +9,5 @@ Ascend NPUs ascend_npu_support_features.md ascend_npu_deepseek_example.md ascend_npu_qwen3_examples.md + ascend_contribution_guide.md ascend_npu_best_practice.md diff --git a/docs/platforms/ascend_npu_support_features.md b/docs/platforms/ascend_npu_support_features.md index fb2cdbfe5..db0b5645c 100644 --- a/docs/platforms/ascend_npu_support_features.md +++ b/docs/platforms/ascend_npu_support_features.md @@ -70,10 +70,10 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--priority-scheduling-`
`preemption-threshold` | `10` | Type: int | **√** | **√** | | `--schedule-conservativeness` | `1.0` | Type: float | **√** | **√** | | `--page-size` | `128` | Type: int | **√** | **√** | +| `--abort-on-priority-`
`when-disabled` | `False` | bool flag
(set to enable) | **√** | **√** | | `--hybrid-kvcache-ratio` | `None` | Optional[float] | **Γ—** | **Γ—** | | `--swa-full-tokens-ratio` | `0.8` | Type: float | **Γ—** | **Γ—** | | `--disable-hybrid-swa-memory` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | -| `--abort-on-priority-`
`when-disabled` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-dynamic-chunking` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | ## Runtime options @@ -83,7 +83,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--device` | `None` | Type: str | **√** | **√** | | `--tensor-parallel-size`
`--tp-size` | `1` | Type: int | **√** | **√** | | `--pipeline-parallel-size`
`--pp-size` | `1` | Type: int | **Γ—** | **Γ—** | -| `--pp-max-micro-batch-size` | `None` | Type: int | **Γ—** | **Γ—** | +| `--pp-max-micro-batch-size` | `None` | Type: int | **√** | **√** | | `--pp-async-batch-depth` | `None` | Type: int | **Γ—** | **Γ—** | | `--stream-interval` | `1` | Type: int | **√** | **√** | | `--stream-output` | `False` | bool flag (set to enable) | **√** | **√** | @@ -91,7 +91,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--constrained-json-`
`whitespace-pattern` | `None` | Type: str | **√** | **√** | | `--constrained-json-`
`disable-any-whitespace` | `False` | bool flag (set to enable) | **√** | **√** | | `--watchdog-timeout` | `300` | Type: float | **√** | **√** | -| `--soft-watchdog-timeout` | `300` | Type: float | **√** | **√** | +| `--soft-watchdog-timeout` | `300` | Type: float | **Γ—** | **Γ—** | +| `--download-dir` | `None` | Type: str | **Γ—** | **Γ—** | | `--dist-timeout` | `None` | Type: int | **√** | **√** | | `--base-gpu-id` | `0` | Type: int | **√** | **√** | | `--gpu-id-step` | `1` | Type: int | **√** | **√** | @@ -106,21 +107,22 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--log-level-http` | `None` | Type: str | **√** | **√** | | `--log-requests` | `False` | bool flag
(set to enable) | **√** | **√** | | `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | **√** | **√** | -| `--log-requests-format` | text | text, json | **√** | **√** | -| `--crash-dump-folder` | `None` | Type: str | **Γ—** | **Γ—** | +| `--log-requests-format` | text | text, json | **Γ—** | **Γ—** | +| `--crash-dump-folder` | `None` | Type: str | **√** | **√** | +| `--show-time-cost` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--crash-on-nan` | `False` | Type: str | **Γ—** | **Γ—** | -| `--enable-metrics` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--enable-metrics-for-`
`all-schedulers` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--tokenizer-metrics-`
`custom-labels-header` | `x-custom-labels` | Type: str | **√** | **√** | -| `--tokenizer-metrics-`
`allowed-custom-labels` | `None` | List[str] | **√** | **√** | -| `--bucket-time-to-`
`first-token` | `None` | List[float] | **√** | **√** | -| `--bucket-inter-token-`
`latency` | `None` | List[float] | **√** | **√** | -| `--bucket-e2e-request-`
`latency` | `None` | List[float] | **√** | **√** | -| `--collect-tokens-`
`histogram` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--prompt-tokens-buckets` | `None` | List[str] | **√** | **√** | -| `--generation-tokens-buckets` | `None` | List[str] | **√** | **√** | -| `--gc-warning-threshold-secs` | `0.0` | Type: float | **√** | **√** | -| `--decode-log-interval` | `40` | Type: int | **√** | **√** | +| `--enable-metrics` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--enable-metrics-for-`
`all-schedulers` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--tokenizer-metrics-`
`custom-labels-header` | `x-custom-labels` | Type: str | **Γ—** | **Γ—** | +| `--tokenizer-metrics-`
`allowed-custom-labels` | `None` | List[str] | **Γ—** | **Γ—** | +| `--bucket-time-to-`
`first-token` | `None` | List[float] | **Γ—** | **Γ—** | +| `--bucket-inter-token-`
`latency` | `None` | List[float] | **Γ—** | **Γ—** | +| `--bucket-e2e-request-`
`latency` | `None` | List[float] | **Γ—** | **Γ—** | +| `--collect-tokens-`
`histogram` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--prompt-tokens-buckets` | `None` | List[str] | **Γ—** | **Γ—** | +| `--generation-tokens-buckets` | `None` | List[str] | **Γ—** | **Γ—** | +| `--gc-warning-threshold-secs` | `0.0` | Type: float | **Γ—** | **Γ—** | +| `--decode-log-interval` | `40` | Type: int | **Γ—** | **Γ—** | | `--enable-request-time-`
`stats-logging` | `False` | bool flag
(set to enable) | **√** | **√** | | `--kv-events-config` | `None` | Type: str | **Γ—** | **Γ—** | | `--enable-trace` | `False` | bool flag
(set to enable) | **√** | **√** | @@ -128,26 +130,26 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen ## RequestMetricsExporter configuration -| Argument | Defaults | Options | A2 | A3 | -|---------------------------------------|----------|--------------------------------|:----------------------------------------:|:----------------------------------------:| +| Argument | Defaults | Options | A2 | A3 | +|---------------------------------------|----------|--------------------------------|:--------------------------------------:|:--------------------------------------:| | `--export-metrics-to-`
`file` | `False` | bool flag
(set to enable) | **√** | **√** | | `--export-metrics-to-`
`file-dir` | `None` | Type: str | **√** | **√** | ## API related -| Argument | Defaults | Options | A2 | A3 | -|-------------------------|-----------|--------------------------------|:----------------------------------------:|:----------------------------------------:| -| `--api-key` | `None` | Type: str | **√** | **√** | -| `--admin-api-key` | `None` | Type: str | **√** | **√** | -| `--served-model-name` | `None` | Type: str | **√** | **√** | -| `--weight-version` | `default` | Type: str | **√** | **√** | -| `--chat-template` | `None` | Type: str | **√** | **√** | -| `--completion-template` | `None` | Type: str | **√** | **√** | -| `--enable-cache-report` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--reasoning-parser` | `None` | `deepseek-r1` | **√** | **√** | -| `--tool-call-parser` | `None` | `llama`,`pythonic` | **√** | **√** | -| `--sampling-defaults` | `model` | `openai`, `model` | **√** | **√** | -| `--tool-server` | `None` | Type: str | **Γ—** | **Γ—** | +| Argument | Defaults | Options | A2 | A3 | +|-------------------------|------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:| +| `--api-key` | `None` | Type: str | **√** | **√** | +| `--served-model-name` | `None` | Type: str | **√** | **√** | +| `--weight-version` | `default` | Type: str | **√** | **√** | +| `--chat-template` | `None` | Type: str | **√** | **√** | +| `--completion-template` | `None` | Type: str | **√** | **√** | +| `--file-storage-path` | `sglang_storage` | Type: str | **√** | **√** | +| `--enable-cache-report` | `False` | bool flag
(set to enable) | **√** | **√** | +| `--reasoning-parser` | `None` | `deepseek-r1` | **√** | **√** | +| `--tool-call-parser` | `None` | `llama`,`pythonic` | **√** | **√** | +| `--sampling-defaults` | `model` | `openai`, `model` | **Γ—** | **Γ—** | +| `--tool-server` | `None` | Type: str | **Γ—** | **Γ—** | ## Data parallelism @@ -155,7 +157,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen |----------------------------------------|---------------|-----------------------------------------------------------|:----------------------------------------:|:----------------------------------------:| | `--data-parallel-size`
`--dp-size` | `1` | Type: int | **√** | **√** | | `--load-balance-method` | `round_robin` | `round_robin`,
`total_requests`,
`total_tokens` | **√** | **√** | -| `--prefill-round-robin-balance` | `False` | bool flag
(set to enable) | **√** | **√** | +| `--load-watch-interval` | `0.1` | Type: float | **Γ—** | **Γ—** | +| `--prefill-round-robin-balance` | `False` | bool flag
(set to enable) | **√** | **√** | ## Multi-node distributed serving @@ -182,7 +185,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--lora-paths` | `None` | Type: List[str] /
JSON objects | **√** | **√** | | `--max-loras-per-batch` | `8` | Type: int | **√** | **√** | | `--max-loaded-loras` | `None` | Type: int | **√** | **√** | -| `--lora-eviction-policy` | `lru` | `lru`,
`fifo` | **√** | **√** | +| `--lora-eviction-policy` | `lru` | `lru`,
`fifo` | **Γ—** | **Γ—** | | `--lora-backend` | `triton` | `triton` | **√** | **√** | | `--max-lora-chunk-size` | `16` | `16`, `32`,
`64`, `128` | **Γ—** | **Γ—** | @@ -212,9 +215,9 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--speculative-num-steps` | `None` | Type: int | **√** | **√** | | `--speculative-eagle-topk` | `None` | Type: int | **√** | **√** | | `--speculative-num-draft-tokens` | `None` | Type: int | **√** | **√** | -| `--speculative-accept-`
`threshold-single` | `1.0` | Type: float | **Γ—** | **Γ—** | -| `--speculative-accept-`
`threshold-acc` | `1.0` | Type: float | **Γ—** | **Γ—** | -| `--speculative-token-map` | `None` | Type: str | **√** | **√** | +| `--speculative-accept-`
`threshold-single` | `1.0` | Type: float | **√** | **√** | +| `--speculative-accept-`
`threshold-acc` | `1.0` | Type: float | **√** | **√** | +| `--speculative-token-map` | `None` | Type: str | **Γ—** | **Γ—** | | `--speculative-attention-`
`mode` | `prefill` | `prefill`,
`decode` | **√** | **√** | | `--speculative-moe-runner-`
`backend` | `None` | `auto` | **√** | **√** | | `--speculative-moe-a2a-`
`backend` | `None` | `ascend_fuseep` | **√** | **√** | @@ -233,30 +236,37 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--speculative-ngram-`
`branch-length` | `18` | Type: int | **Γ—** | **Γ—** | | `--speculative-ngram-`
`capacity` | `10000000` | Type: int | **Γ—** | **Γ—** | +## Multi-layer Eagle speculative decoding + +| Argument | Defaults | Options | A2 | A3 | +|---------------------------------------------------|----------|-------------------------------------|:----------------------------------------:|:----------------------------------------:| +| `--enable-multi-layer-eagle` | `False` | Bool flag
(set to enable) | **Γ—** | **Γ—** | + ## Expert parallelism -| Argument | Defaults | Options | A2 | A3 | -|-------------------------------------------------------|-----------|---------------------------------------------|:-----------------------------------------:|:----------------------------------------:| -| `--expert-parallel-size`
`--ep-size`
`--ep` | `1` | Type: int | **√** | **√** | -| `--moe-a2a-backend` | `none` | `none`,
`deepep`,
`ascend_fuseep` | **√** | **√** | -| `--moe-runner-backend` | `auto` | `auto`, `triton` | **√** | **√** | -| `--flashinfer-mxfp4-`
`moe-precision` | `default` | `default`,
`bf16` | **Γ—** | **Γ—** | -| `--enable-flashinfer-`
`allreduce-fusion` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | -| `--deepep-mode` | `auto` | `normal`,
`low_latency`,
`auto` | **√** | **√** | -| `--deepep-config` | `None` | Type: str | **Γ—** | **Γ—** | -| `--ep-num-redundant-experts` | `0` | Type: int | **√** | **√** | -| `--ep-dispatch-algorithm` | `None` | Type: str | **√** | **√** | -| `--init-expert-location` | `trivial` | Type: str | **√** | **√** | -| `--enable-eplb` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--eplb-algorithm` | `auto` | Type: str | **√** | **√** | -| `--eplb-rebalance-layers-`
`per-chunk` | `None` | Type: int | **√** | **√** | -| `--eplb-min-rebalancing-`
`utilization-threshold` | `1.0` | Type: float | **√** | **√** | -| `--expert-distribution-`
`recorder-mode` | `None` | Type: str | **√** | **√** | -| `--expert-distribution-`
`recorder-buffer-size` | `None` | Type: int | **√** | **√** | -| `--enable-expert-distribution-`
`metrics` | `False` | bool flag
(set to enable) | ***√** | **√** | -| `--moe-dense-tp-size` | `None` | Type: int | **√** | **√** | -| `--elastic-ep-backend` | `None` | `none`, `mooncake` | **Γ—** | **Γ—** | -| `--mooncake-ib-device` | `None` | Type: str | **Γ—** | **Γ—** | +| Argument | Defaults | Options | A2 | A3 | +|-------------------------------------------------------|-----------|---------------------------------------------|:----------------------------------------:|:----------------------------------------:| +| `--expert-parallel-size`
`--ep-size`
`--ep` | `1` | Type: int | **√** | **√** | +| `--moe-a2a-backend` | `none` | `none`,
`deepep`,
`ascend_fuseep` | **√** | **√** | +| `--moe-runner-backend` | `auto` | `auto`, `triton` | **√** | **√** | +| `--flashinfer-mxfp4-`
`moe-precision` | `default` | `default`,
`bf16` | **Γ—** | **Γ—** | +| `--enable-flashinfer-`
`allreduce-fusion` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--deepep-mode` | `auto` | `normal`,
`low_latency`,
`auto` | **√** | **√** | +| `--deepep-config` | `None` | Type: str | **Γ—** | **Γ—** | +| `--ep-num-redundant-experts` | `0` | Type: int | **Γ—** | **Γ—** | +| `--ep-dispatch-algorithm` | `None` | Type: str | **Γ—** | **Γ—** | +| `--init-expert-location` | `trivial` | Type: str | **Γ—** | **Γ—** | +| `--enable-eplb` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--eplb-algorithm` | `auto` | Type: str | **Γ—** | **Γ—** | +| `--eplb-rebalance-num-iterations` | `1000` | Type: int | **Γ—** | **Γ—** | +| `--eplb-rebalance-layers-`
`per-chunk` | `None` | Type: int | **Γ—** | **Γ—** | +| `--eplb-min-rebalancing-`
`utilization-threshold` | `1.0` | Type: float | **Γ—** | **Γ—** | +| `--expert-distribution-`
`recorder-mode` | `None` | Type: str | **Γ—** | **Γ—** | +| `--expert-distribution-`
`recorder-buffer-size` | `None` | Type: int | **Γ—** | **Γ—** | +| `--enable-expert-distribution-`
`metrics` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--moe-dense-tp-size` | `None` | Type: int | **√** | **√** | +| `--elastic-ep-backend` | `None` | `none`, `mooncake` | **Γ—** | **Γ—** | +| `--mooncake-ib-device` | `None` | Type: str | **Γ—** | **Γ—** | ## Mamba Cache @@ -300,6 +310,12 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--kt-num-gpu-experts` | `None` | Type: int | **Γ—** | **Γ—** | | `--kt-max-deferred-`
`experts-per-token` | `None` | Type: int | **Γ—** | **Γ—** | +## Diffusion LLM +| Argument | Defaults | Options | A2 | A3 | +|----------------------------------------------|-----------|-----------|:--------------------------------------:|:--------------------------------------:| +| `--dllm-algorithm` | `None` | Type: str | **Γ—** | **Γ—** | +| `--dllm-algorithm-config` | `None` | Type: str | **Γ—** | **Γ—** | + ## Double Sparsity | Argument | Defaults | Options | A2 | A3 | @@ -324,8 +340,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen ## Args for multi-item scoring | Argument | Defaults | Options | A2 | A3 | -|----------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:| -| `--multi-item-scoring-delimiter` | `None` | Type: int | **Γ—** | **Γ—** | +|----------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:| +| `--multi-item-scoring-delimiter` | `None` | Type: int | **√** | **√** | ## Optimization/debug options @@ -337,13 +353,13 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--disable-cuda-graph` | `False` | bool flag
(set to enable) | **√** | **√** | | `--disable-cuda-graph-`
`padding` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-profile-`
`cuda-graph` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--enable-cudagraph-gc` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--enable-cudagraph-gc` | `False` | bool flag
(set to enable) | **Γ—** | **√** | | `--enable-nccl-nvls` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--enable-symm-mem` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--disable-flashinfer-`
`cutlass-moe-fp4-allgather` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--enable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--disable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--disable-outlines-`
`disk-cache` | `False` | bool flag
(set to enable) | **√** | **√** | +| `--disable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--disable-outlines-`
`disk-cache` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--disable-custom-`
`all-reduce` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-mscclpp` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--enable-torch-`
`symm-mem` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | @@ -352,8 +368,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--enable-dp-attention` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-dp-lm-head` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-two-`
`batch-overlap` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | -| `--enable-single-`
`batch-overlap` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | -| `--tbo-token-`
`distribution-threshold` | `0.48` | Type: float | **Γ—** | **Γ—** | +| `--enable-single-`
`batch-overlap` | `False` | bool flag
(set to enable) | **√** | **√** | +| `--tbo-token-`
`distribution-threshold` | `0.48` | Type: float | **√** | **√** | | `--enable-torch-`
`compile` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-torch-`
`compile-debug-mode` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-piecewise-`
`cuda-graph` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | @@ -362,26 +378,26 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--torch-compile-max-bs` | `32` | Type: int | **Γ—** | **Γ—** | | `--piecewise-cuda-`
`graph-max-tokens` | `4096` | Type: int | **Γ—** | **Γ—** | | `--torchao-config` | `` | Type: str | **Γ—** | **Γ—** | -| `--enable-nan-detection` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--enable-nan-detection` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-p2p-check` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--triton-attention-`
`reduce-in-fp32` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--triton-attention-`
`num-kv-splits` | `8` | Type: int | **Γ—** | **Γ—** | | `--triton-attention-`
`split-tile-size` | `None` | Type: int | **Γ—** | **Γ—** | | `--num-continuous-`
`decode-steps` | `1` | Type: int | **Γ—** | **Γ—** | -| `--delete-ckpt-`
`after-loading` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--delete-ckpt-`
`after-loading` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-memory-saver` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--enable-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--enable-draft-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--allow-auto-truncate` | `False` | bool flag
(set to enable) | **√** | **√** | | `--enable-custom-`
`logit-processor` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--flashinfer-mla-`
`disable-ragged` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | -| `--disable-shared-`
`experts-fusion` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | -| `--disable-chunked-`
`prefix-cache` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--disable-shared-`
`experts-fusion` | `False` | bool flag
(set to enable) | **√** | **√** | +| `--disable-chunked-`
`prefix-cache` | `False` | bool flag
(set to enable) | **√** | **√** | | `--disable-fast-`
`image-processor` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--keep-mm-feature-`
`on-device` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--enable-return-`
`hidden-states` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--enable-return-`
`routed-experts` | `False` | bool flag
(set to enable) | **√** | **√** | -| `--scheduler-recv-`
`interval` | `1` | Type: int | **Γ—** | **Γ—** | +| `--enable-return-`
`routed-experts` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | +| `--scheduler-recv-`
`interval` | `1` | Type: int | **√** | **√** | | `--numa-node` | `None` | List[int] | **Γ—** | **Γ—** | | `--rl-on-policy-target` | `None` | `fsdp` | **Γ—** | **Γ—** | | `--enable-layerwise-`
`nvtx-marker` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | @@ -434,7 +450,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | Argument | Defaults | Options | A2 | A3 | |-------------------------------------------------------------------------|----------|---------------------------------|:----------------------------------------:|------------------------------------------| | `--custom-weight-loader` | `None` | List[str] | **Γ—** | **Γ—** | -| `--weight-loader-disable-`
`mmap` | `False` | bool flag
(set to enable) | **√** | **√** | +| `--weight-loader-disable-`
`mmap` | `False` | bool flag
(set to enable) | **Γ—** | **Γ—** | | `--remote-instance-weight-`
`loader-seed-instance-ip` | `None` | Type: str | **Γ—** | **Γ—** | | `--remote-instance-weight-`
`loader-seed-instance-service-port` | `None` | Type: int | **Γ—** | **Γ—** | | `--remote-instance-weight-`
`loader-send-weights-group-ports` | `None` | Type: JSON
list | **Γ—** | **Γ—** | diff --git a/docs/platforms/ascend_npu_support_models.md b/docs/platforms/ascend_npu_support_models.md index 1ecc3c48f..2c5c4c9a9 100644 --- a/docs/platforms/ascend_npu_support_models.md +++ b/docs/platforms/ascend_npu_support_models.md @@ -22,6 +22,7 @@ You are welcome to enable various models based on your business requirements. | vllm-ascend/QWQ-32B-W8A8 | Qwen | **√** | **√** | | meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama | **√** | **√** | | AI-ModelScope/Llama-3.1-8B-Instruct | Llama | **√** | **√** | +| LLM-Research/llama-2-7b | Llama | **√** | **√** | | LLM-Research/Llama-3.2-1B-Instruct | Llama | **√** | **√** | | mistralai/Mistral-7B-Instruct-v0.2 | Mistral | **√** | **√** | | google/gemma-3-4b-it | Gemma | **√** | **√** | @@ -42,11 +43,12 @@ You are welcome to enable various models based on your business requirements. | inclusionAI/Ling-lite | Ling | **√** | **√** | | ibm-granite/granite-3.1-8b-instruct | Granite | **√** | **√** | | ibm-granite/granite-3.0-3b-a800m-instruct | Granite MoE | **√** | **√** | -| databricks/dbrx-instruct | DBRX (Databricks) | **√** | **√** | +| AI-ModelScope/dbrx-instruct | DBRX (Databricks) | **√** | **√** | | baichuan-inc/Baichuan2-13B-Chat | Baichuan 2 (7B, 13B) | **√** | **√** | | baidu/ERNIE-4.5-21B-A3B-PT | ERNIE-4.5 (4.5, 4.5MoE series) | **√** | **√** | | openbmb/MiniCPM3-4B | MiniCPM (v3, 4B) | **√** | **√** | -| openai/gpt-oss-120b | GPTOSS | **√** | **√** | +| Kimi/Kimi-K2-Think | Kimi | **√** | **√** | +| openai/gpt-oss-120b | GPTOSS | **Γ—** | **Γ—** | ## Multimodal Language Models @@ -59,8 +61,10 @@ You are welcome to enable various models based on your business requirements. | Qwen/Qwen3-VL-4B-Instruct | Qwen-VL | **√** | **√** | | Qwen/Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **√** | **√** | | deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **√** | **√** | +| deepseek-ai/Janus-Pro-1B | Janus-Pro (1B, 7B) | **√** | **√** | | deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **√** | **√** | | openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **√** | **√** | +| openbmb/MiniCPM-o-2_6 | MiniCPM-V / MiniCPM-o | **√** | **√** | | google/gemma-3-4b-it | Gemma 3 (Multimodal) | **√** | **√** | | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.1-24B | **√** | **√** | | microsoft/Phi-4-multimodal-instruct | Phi-4-multimodal-instruct | **√** | **√** | @@ -90,7 +94,7 @@ You are welcome to enable various models based on your business requirements. | Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **√** | **√** | | Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **√** | **√** | | Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **√** | **√** | -| jason9693/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **√** | **√** | +| Howeee/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **√** | **√** | | Skywork/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **√** | **√** | ## Rerank Models