Evaluation And Report Generation
This folder owns benchmark evaluation, shard merging, and report/summary generation.
It does not train models and does not build model checkpoints.
Main Files
eval_tokenizer_swap_benchmark.py
merge_tokenizer_swap_benchmark_shards.py
summarize_heldout_capabilities.py
run_public_heldout_eval_8gpu.sh
Input
The default benchmark is the latest public heldout 2K set kept in dataset_building/:
dataset_building/heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
The evaluation script expects a Hugging Face checkpoint directory as MODEL.
Run
ROOT=/ssd/yi/Tokenizer_Swap \
MODEL=/path/to/checkpoint \
LABEL=my_model \
bash evaluation_reporting/run_public_heldout_eval_8gpu.sh
The wrapper launches one shard per GPU, then merges shard outputs.
Output
Generated outputs go under:
evaluation_reporting/outputs/
This directory is ignored by git.
Each evaluation writes per-item JSONL plus summary JSON. The key reported metrics are:
- MCQ accuracy with average-normalized choice logprob
- MCQ accuracy with summed choice logprob
- perplexity
- NLL per token
- bits per byte
Final Public Heldout 2K Reference
| Model | MCQ acc avg-norm | MCQ acc sum | PPL | NLL/token |
|---|---|---|---|---|
| Native Qwen3-0.6B tokenizer baseline | 0.2960 | 0.2510 | 61.08 | 3.5399 |
| Remap v2, no training | 0.2940 | 0.2410 | 313.62 | 4.6920 |
| Remap v2 + CPT 1B | 0.3005 | 0.2540 | 71.90 | 3.6580 |
| Remap v2 + CPT 5B | 0.3020 | 0.2615 | 66.95 | 3.5806 |
| Remap v2 + SFT 1M | 0.3105 | 0.2590 | 114.75 | 3.9400 |
| Remap v2 + SFT 1M + v4 continuation | 0.3165 | 0.2595 | 117.33 | 3.9755 |
| Remap v2 + CPT 5B + SFT 1M | 0.3280 | 0.2740 | 88.76 | 3.7899 |