# Evaluation And Report Generation

This folder owns benchmark evaluation, shard merging, and report/summary generation.

It does not train models and does not build model checkpoints.

## Main Files

```text
eval_tokenizer_swap_benchmark.py
merge_tokenizer_swap_benchmark_shards.py
summarize_heldout_capabilities.py
run_public_heldout_eval_8gpu.sh
```

## Input

The default benchmark is the latest public heldout 2K set kept in `dataset_building/`:

```text
dataset_building/heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
```

The evaluation script expects a Hugging Face checkpoint directory as `MODEL`.

## Run

```bash
ROOT=/ssd/yi/Tokenizer_Swap \
MODEL=/path/to/checkpoint \
LABEL=my_model \
bash evaluation_reporting/run_public_heldout_eval_8gpu.sh
```

The wrapper launches one shard per GPU, then merges shard outputs.

## Output

Generated outputs go under:

```text
evaluation_reporting/outputs/
```

This directory is ignored by git.

Each evaluation writes per-item JSONL plus summary JSON. The key reported metrics are:

- MCQ accuracy with average-normalized choice logprob
- MCQ accuracy with summed choice logprob
- perplexity
- NLL per token
- bits per byte

## Final Public Heldout 2K Reference

| Model | MCQ acc avg-norm | MCQ acc sum | PPL | NLL/token |
|---|---:|---:|---:|---:|
| Native Qwen3-0.6B tokenizer baseline | 0.2960 | 0.2510 | 61.08 | 3.5399 |
| Remap v2, no training | 0.2940 | 0.2410 | 313.62 | 4.6920 |
| Remap v2 + CPT 1B | 0.3005 | 0.2540 | 71.90 | 3.6580 |
| Remap v2 + CPT 5B | 0.3020 | 0.2615 | 66.95 | 3.5806 |
| Remap v2 + SFT 1M | 0.3105 | 0.2590 | 114.75 | 3.9400 |
| Remap v2 + SFT 1M + v4 continuation | 0.3165 | 0.2595 | 117.33 | 3.9755 |
| Remap v2 + CPT 5B + SFT 1M | 0.3280 | 0.2740 | 88.76 | 3.7899 |