chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Doc] refine spec decode docs for SpecV2/STANDALONE/NGRAM (#18321)

Browse Source
This commit is contained in:
shuwenn
2026-02-06 04:12:33 +08:00
committed by GitHub
parent 6a4b81e2d9
commit 8b21dd4b77
1 changed files with 202 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

203
docs/advanced_features/speculative_decoding.ipynb
View File

@@ -27,7 +27,7 @@
"## EAGLE Decoding\n",
"\n",
"To enable EAGLE speculative decoding the following parameters are relevant:\n",
"* `speculative_draft_model_path`: Specifies draft model. This parameter is required.\n",
"* `speculative_draft_model_path`: Draft model path/weights. **Typically required** for EAGLE/EAGLE3 and STANDALONE. For some MTP-enabled models, this can be omitted (SGLang may auto-handle/auto-fill it).\n",
"* `speculative_num_steps`: Depth of autoregressive drafting. Increases speculation range but risks rejection cascades. Default is 5.\n",
"* `speculative_eagle_topk`: Branching factor per step. Improves candidate diversity, will lead to higher acceptance rate, but more lead to higher memory/compute consumption. Default is 4.\n",
"* `speculative_num_draft_tokens`: Maximum parallel verification capacity. Allows deeper tree evaluation but will lead to higher GPU memory usage. Default is 8.\n",
@@ -332,6 +332,207 @@
"terminate_process(server_process)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Standalone Speculative Decoding (Small Draft Model)\n",
"\n",
"Besides EAGLE/MTP, SGLang also supports **token-level speculative decoding** using a smaller **draft model**. Enable it with `--speculative-algorithm STANDALONE` and provide a draft model via `--speculative-draft-model-path`.\n",
"\n",
"Relevant parameters:\n",
"- `--speculative-draft-model-path`: Draft model weights (smaller than the target model).\n",
"- `--speculative-num-steps`: Draft depth (how many steps the draft model runs autoregressively).\n",
"- `--speculative-eagle-topk`: Branching factor (token candidates per step).\n",
"- `--speculative-num-draft-tokens`: Verification capacity.\n",
"\n",
"Note:\n",
"- Standalone speculative decoding currently **does not support** `--enable-dp-attention`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"server_process, port = launch_server_cmd(\n",
" \"\"\"\n",
"python3 -m sglang.launch_server --model Qwen/Qwen2.5-7B-Instruct --speculative-algorithm STANDALONE \\\n",
" --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \\\n",
" --speculative-num-steps 4 --speculative-eagle-topk 2 --speculative-num-draft-tokens 7 \\\n",
" --cuda-graph-max-bs 8 --mem-fraction-static 0.7 --log-level warning\n",
"\"\"\"\n",
")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
"\n",
"response = client.chat.completions.create(\n",
" model=\"Qwen/Qwen2.5-7B-Instruct\",\n",
" messages=[\n",
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
" ],\n",
" temperature=0,\n",
" max_tokens=64,\n",
")\n",
"\n",
"print_highlight(f\"Response: {response}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"terminate_process(server_process)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Speculative Decoding V2 (Overlap Scheduler)\n",
"\n",
"SGLang provides an **experimental Speculative Decoding V2** implementation that enables an overlap scheduler and uses V2 speculative workers (e.g. `StandaloneWorkerV2`, `EAGLEWorkerV2`).\n",
"\n",
"To enable it, set the environment variable:\n",
"- `SGLANG_ENABLE_SPEC_V2=True`\n",
"\n",
"Notes:\n",
"- SpecV2 currently only supports `--speculative-eagle-topk 1`. When SpecV2 is enabled, **set `--speculative-eagle-topk 1` explicitly**.\n",
"- If you explicitly set `--speculative-eagle-topk > 1`, the server will error. If you omit `--speculative-eagle-topk`, auto-tuning may pick `topk > 1` for some models (e.g. Llama), which is not supported by SpecV2.\n",
"- This applies to `EAGLE`, `EAGLE3`, and `STANDALONE`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"server_process, port = launch_server_cmd(\n",
" \"\"\"\n",
"SGLANG_ENABLE_SPEC_V2=True python3 -m sglang.launch_server --model Qwen/Qwen2.5-7B-Instruct --speculative-algorithm STANDALONE \\\n",
" --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \\\n",
" --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \\\n",
" --cuda-graph-max-bs 8 --mem-fraction-static 0.7 --log-level warning\n",
"\"\"\"\n",
")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
"\n",
"response = client.chat.completions.create(\n",
" model=\"Qwen/Qwen2.5-7B-Instruct\",\n",
" messages=[\n",
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
" ],\n",
" temperature=0,\n",
" max_tokens=64,\n",
")\n",
"\n",
"print_highlight(f\"Response: {response}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"terminate_process(server_process)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Ngram Speculative Decoding\n",
"\n",
"SGLang also supports **ngram-based speculative decoding** (no separate draft model). It retrieves draft tokens from an ngram cache built from previously generated tokens, and then verifies them with the target model.\n",
"\n",
"Enable it with:\n",
"- `--speculative-algorithm NGRAM`\n",
"\n",
"Common parameters:\n",
"- `--speculative-num-draft-tokens`: Number of draft tokens verified per step.\n",
"- `--speculative-ngram-min-match-window-size` / `--speculative-ngram-max-match-window-size`: Matching window range.\n",
"- `--speculative-ngram-min-bfs-breadth` / `--speculative-ngram-max-bfs-breadth`: BFS breadth range.\n",
"- `--speculative-ngram-branch-length`: How many recent tokens to insert into the cache.\n",
"- `--speculative-ngram-capacity`: Cache capacity.\n",
"\n",
"Notes:\n",
"- Ngram speculative decoding **only supports CUDA**.\n",
"- It currently **does not support** `--enable-dp-attention`.\n",
"- It disables the overlap scheduler and mixed chunked prefill.\n",
"- Optional: set `SGLANG_NGRAM_FORCE_GREEDY_VERIFY=True` to force greedy verification.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"server_process, port = launch_server_cmd(\n",
" \"\"\"\n",
"python3 -m sglang.launch_server --model Qwen/Qwen2.5-7B-Instruct --speculative-algorithm NGRAM \\\n",
" --speculative-num-draft-tokens 16 \\\n",
" --speculative-ngram-max-match-window-size 12 --speculative-ngram-max-bfs-breadth 10 \\\n",
" --cuda-graph-max-bs 8 --mem-fraction-static 0.8 --log-level warning\n",
"\"\"\"\n",
")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
"\n",
"response = client.chat.completions.create(\n",
" model=\"Qwen/Qwen2.5-7B-Instruct\",\n",
" messages=[\n",
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
" ],\n",
" temperature=0,\n",
" max_tokens=64,\n",
")\n",
"\n",
"print_highlight(f\"Response: {response}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"terminate_process(server_process)"
]
},
{
"cell_type": "markdown",
"metadata": {},