[Doc] Add tip on how to use Spec V2 (#15455)
This commit is contained in:
@@ -59,6 +59,24 @@ python -m sglang.launch_server ... --tool-server ip-1:port-1,ip-2:port-2
|
||||
```
|
||||
The URLs should be MCP SSE servers that expose server information and well-documented tools. These tools are added to the system prompt so the model can use them.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
SGLang supports speculative decoding for GPT-OSS models using EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes.
|
||||
|
||||
**Usage**:
|
||||
Add `--speculative-algorithm EAGLE3` along with the draft model path.
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
```{tip}
|
||||
To enable the experimental overlap scheduler for EAGLE3 speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
```
|
||||
|
||||
### Quick Demo
|
||||
|
||||
```python
|
||||
|
||||
Reference in New Issue
Block a user