chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Spec][Ngram] 1/N: Reference based Speculative Decoding refactor (#20393)

Browse Source
This commit is contained in:
kpham-sgl
2026-03-22 00:55:10 -07:00
committed by GitHub
parent d9f5c2179c
commit 6d160b42bb
13 changed files with 1039 additions and 388 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/server_arguments.md
+1 -1
View File
@@ -298,7 +298,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
| `--speculative-ngram-max-match-window-size` | The maximum window size for pattern matching in ngram speculative decoding. | `12` | Type: int |
| `--speculative-ngram-min-bfs-breadth` | The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding. | `1` | Type: int |
| `--speculative-ngram-max-bfs-breadth` | The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding. | `10` | Type: int |
| `--speculative-ngram-match-type` | The match type for cache tree. | `BFS` | `BFS`, `PROB` |
| `--speculative-ngram-match-type` | Ngram tree-building mode. `BFS` selects recency-based expansion and `PROB` selects frequency-based expansion. This setting is forwarded to the ngram cache implementation. | `BFS` | `BFS`, `PROB` |
| `--speculative-ngram-branch-length` | The branch length for ngram speculative decoding. | `18` | Type: int |
| `--speculative-ngram-capacity` | The cache capacity for ngram speculative decoding. | `10000000` | Type: int |
docs/advanced_features/speculative_decoding.md
+2 -2
View File
@@ -392,7 +392,7 @@ Enable it with:
| `--speculative-ngram-max-match-window-size` | Maximum matching window size. | `12` |
| `--speculative-ngram-min-bfs-breadth` | Minimum BFS breadth. | `1` |
| `--speculative-ngram-max-bfs-breadth` | Maximum BFS breadth. | `10` |
| `--speculative-ngram-match-type` | Match type: `"BFS"` or `"PROB"`. | `"BFS"` |
| `--speculative-ngram-match-type` | Ngram tree-building mode: `"BFS"` for recency-based expansion or `"PROB"` for frequency-based expansion. | `"BFS"` |
| `--speculative-ngram-branch-length` | How many recent tokens to insert into the cache. | `18` |
| `--speculative-ngram-capacity` | Cache capacity (number of entries). | `10,000,000` |
@@ -468,7 +468,7 @@ Below is a comprehensive list of all speculative decoding parameters available i
| `--speculative-ngram-max-match-window-size` | `int` | `12` | Maximum ngram matching window |
| `--speculative-ngram-min-bfs-breadth` | `int` | `1` | Minimum BFS breadth |
| `--speculative-ngram-max-bfs-breadth` | `int` | `10` | Maximum BFS breadth |
| `--speculative-ngram-match-type` | `str` | `"BFS"` | Match type: `"BFS"` or `"PROB"` |
| `--speculative-ngram-match-type` | `str` | `"BFS"` | Ngram tree-building mode: `"BFS"` for recency-based expansion or `"PROB"` for frequency-based expansion |
| `--speculative-ngram-branch-length` | `int` | `18` | Recent tokens to insert into cache |
| `--speculative-ngram-capacity` | `int` | `10,000,000` | Cache capacity |
docs/platforms/ascend_npu_support_features.md
+1 -1
View File
@@ -228,7 +228,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
| `--speculative-ngram-`<br/>`max-match-window-size` | `12` | Type: int | Experimental |
| `--speculative-ngram-`<br/>`min-bfs-breadth` | `1` | Type: int | Experimental |
| `--speculative-ngram-`<br/>`max-bfs-breadth` | `10` | Type: int | Experimental |
| `--speculative-ngram-`<br/>`match-type` | `BFS` | `BFS`,<br/> `PROB` | Experimental |
| `--speculative-ngram-`<br/>`match-type` | `BFS` | `BFS`,<br/> `PROB` | Experimental. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion. |
| `--speculative-ngram-`<br/>`branch-length` | `18` | Type: int | Experimental |
| `--speculative-ngram-`<br/>`capacity` | `10000000` | Type: int | Experimental |