chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Add top-p to run_eval.py (#16844)

Browse Source
This commit is contained in:
hlu1
2026-01-10 01:10:37 -08:00
committed by GitHub
parent 9fd2358cc2
commit aeb480c11f
3 changed files with 17 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/basic_usage/deepseek_v32.md
View File

@@ -190,15 +190,25 @@ Output throughput: 4418.617 token/s
Accuracy benchmark on long context can be tested on GPQA-diamond dataset with long output tokens and thinking enabled:
```bash
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 120000 --repeat 8 --thinking-mode deepseek-v3
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --thinking-mode deepseek-v3
```
The mean accuracy over 8 runs shows 0.797, which matches the number 79.9 in official tech report.
The mean accuracy over 8 runs shows 0.797, which matches the number 0.799 in official tech report.
```bash
Repeat: 8, mean: 0.797
Scores: ['0.808', '0.798', '0.808', '0.798', '0.783', '0.788', '0.803', '0.793']
```
For Deepseek V3.2, Deepseek recommends setting the sampling parameters to temperature = 1.0, top_p = 0.95:
```bash
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --top-p 0.95 --temperature 1.0 --thinking-mode deepseek-v3
Repeat: 8, mean: 0.840
Scores: ['0.848', '0.808', '0.848', '0.838', '0.879', '0.813', '0.838', '0.848']
```
which matches the official score, 0.824, as reported in the [Deepseek-V3.2 technical report](https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/assets/paper.pdf).
### Accuracy Test with `aime 2025`
Prepare the environment by installing NeMo-Skills in the docker or your own virtual environment: