[Doc] Add tip on how to use Spec V2 (#15455)
This commit is contained in:
@@ -193,6 +193,10 @@ To enable DeepSeek MTP for large batch sizes (>48), you need to adjust some para
|
||||
- Set `--cuda-graph-bs`. It's a list of batch sizes for cuda graph capture. The [default captured batch sizes for speculative decoding](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/server_args.py#L888-L895) is 48. You can customize this by including more batch sizes.
|
||||
```
|
||||
|
||||
```{tip}
|
||||
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
```
|
||||
|
||||
|
||||
### Reasoning Content for DeepSeek R1 & V3.1
|
||||
|
||||
|
||||
@@ -6,11 +6,6 @@ To serve GLM-4.5 / GLM-4.6 FP8 models on 8xH100/H200 GPUs:
|
||||
python3 -m sglang.launch_server --model zai-org/GLM-4.6-FP8 --tp 8
|
||||
```
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests
|
||||
capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.
|
||||
|
||||
### EAGLE Speculative Decoding
|
||||
|
||||
**Description**: SGLang has supported GLM-4.5 / GLM-4.6 models
|
||||
@@ -35,9 +30,12 @@ python3 -m sglang.launch_server \
|
||||
--enable-custom-logit-processor
|
||||
```
|
||||
|
||||
**Note**: For GLM-4.7, `--tool-call-parser` should be set to `glm47`, for GLM-4.5 and GLM-4.6, it should be set to `glm45`.
|
||||
```{tip}
|
||||
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
```
|
||||
|
||||
### Thinking Budget
|
||||
### Thinking Budget for GLM-4.5 / GLM-4.6
|
||||
**Note**: For GLM-4.7, `--tool-call-parser` should be set to `glm47`, for GLM-4.5 and GLM-4.6, it should be set to `glm45`.
|
||||
|
||||
In SGLang, we can implement thinking budget with `CustomLogitProcessor`.
|
||||
|
||||
|
||||
@@ -59,6 +59,24 @@ python -m sglang.launch_server ... --tool-server ip-1:port-1,ip-2:port-2
|
||||
```
|
||||
The URLs should be MCP SSE servers that expose server information and well-documented tools. These tools are added to the system prompt so the model can use them.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
SGLang supports speculative decoding for GPT-OSS models using EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes.
|
||||
|
||||
**Usage**:
|
||||
Add `--speculative-algorithm EAGLE3` along with the draft model path.
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
```{tip}
|
||||
To enable the experimental overlap scheduler for EAGLE3 speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
```
|
||||
|
||||
### Quick Demo
|
||||
|
||||
```python
|
||||
|
||||
Reference in New Issue
Block a user