[Doc] Add tip on how to use Spec V2 (#15455)

This commit is contained in:
b8zhong
2026-01-15 13:30:18 -08:00
committed by GitHub
parent 7dde3438e2
commit 3d72944fb8
3 changed files with 27 additions and 7 deletions

View File

@@ -193,6 +193,10 @@ To enable DeepSeek MTP for large batch sizes (>48), you need to adjust some para
- Set `--cuda-graph-bs`. It's a list of batch sizes for cuda graph capture. The [default captured batch sizes for speculative decoding](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/server_args.py#L888-L895) is 48. You can customize this by including more batch sizes.
```
```{tip}
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
```
### Reasoning Content for DeepSeek R1 & V3.1

View File

@@ -6,11 +6,6 @@ To serve GLM-4.5 / GLM-4.6 FP8 models on 8xH100/H200 GPUs:
python3 -m sglang.launch_server --model zai-org/GLM-4.6-FP8 --tp 8
```
### Configuration Tips
- `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests
capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.
### EAGLE Speculative Decoding
**Description**: SGLang has supported GLM-4.5 / GLM-4.6 models
@@ -35,9 +30,12 @@ python3 -m sglang.launch_server \
--enable-custom-logit-processor
```
**Note**: For GLM-4.7, `--tool-call-parser` should be set to `glm47`, for GLM-4.5 and GLM-4.6, it should be set to `glm45`.
```{tip}
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
```
### Thinking Budget
### Thinking Budget for GLM-4.5 / GLM-4.6
**Note**: For GLM-4.7, `--tool-call-parser` should be set to `glm47`, for GLM-4.5 and GLM-4.6, it should be set to `glm45`.
In SGLang, we can implement thinking budget with `CustomLogitProcessor`.

View File

@@ -59,6 +59,24 @@ python -m sglang.launch_server ... --tool-server ip-1:port-1,ip-2:port-2
```
The URLs should be MCP SSE servers that expose server information and well-documented tools. These tools are added to the system prompt so the model can use them.
## Speculative Decoding
SGLang supports speculative decoding for GPT-OSS models using EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes.
**Usage**:
Add `--speculative-algorithm EAGLE3` along with the draft model path.
```bash
python3 -m sglang.launch_server \
--model-path openai/gpt-oss-120b \
--speculative-algorithm EAGLE3 \
--speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \
--tp 2
```
```{tip}
To enable the experimental overlap scheduler for EAGLE3 speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
```
### Quick Demo
```python