66 lines
1.7 KiB
Markdown
66 lines
1.7 KiB
Markdown
# Evaluation And Report Generation
|
|
|
|
This folder owns benchmark evaluation, shard merging, and report/summary generation.
|
|
|
|
It does not train models and does not build model checkpoints.
|
|
|
|
## Main Files
|
|
|
|
```text
|
|
eval_tokenizer_swap_benchmark.py
|
|
merge_tokenizer_swap_benchmark_shards.py
|
|
summarize_heldout_capabilities.py
|
|
run_public_heldout_eval_8gpu.sh
|
|
```
|
|
|
|
## Input
|
|
|
|
The default benchmark is the latest public heldout 2K set kept in `dataset_building/`:
|
|
|
|
```text
|
|
dataset_building/heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
|
|
```
|
|
|
|
The evaluation script expects a Hugging Face checkpoint directory as `MODEL`.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
ROOT=/ssd/yi/Tokenizer_Swap \
|
|
MODEL=/path/to/checkpoint \
|
|
LABEL=my_model \
|
|
bash evaluation_reporting/run_public_heldout_eval_8gpu.sh
|
|
```
|
|
|
|
The wrapper launches one shard per GPU, then merges shard outputs.
|
|
|
|
## Output
|
|
|
|
Generated outputs go under:
|
|
|
|
```text
|
|
evaluation_reporting/outputs/
|
|
```
|
|
|
|
This directory is ignored by git.
|
|
|
|
Each evaluation writes per-item JSONL plus summary JSON. The key reported metrics are:
|
|
|
|
- MCQ accuracy with average-normalized choice logprob
|
|
- MCQ accuracy with summed choice logprob
|
|
- perplexity
|
|
- NLL per token
|
|
- bits per byte
|
|
|
|
## Final Public Heldout 2K Reference
|
|
|
|
| Model | MCQ acc avg-norm | MCQ acc sum | PPL | NLL/token |
|
|
|---|---:|---:|---:|---:|
|
|
| Native Qwen3-0.6B tokenizer baseline | 0.2960 | 0.2510 | 61.08 | 3.5399 |
|
|
| Remap v2, no training | 0.2940 | 0.2410 | 313.62 | 4.6920 |
|
|
| Remap v2 + CPT 1B | 0.3005 | 0.2540 | 71.90 | 3.6580 |
|
|
| Remap v2 + CPT 5B | 0.3020 | 0.2615 | 66.95 | 3.5806 |
|
|
| Remap v2 + SFT 1M | 0.3105 | 0.2590 | 114.75 | 3.9400 |
|
|
| Remap v2 + SFT 1M + v4 continuation | 0.3165 | 0.2595 | 117.33 | 3.9755 |
|
|
| Remap v2 + CPT 5B + SFT 1M | 0.3280 | 0.2740 | 88.76 | 3.7899 |
|