# Tokenizer Swap

This is the cleaned migration of the tokenizer-swap experiments. The repo is organized by workflow boundary, not by experiment date or temporary artifact type.

It keeps only the final useful recipes:

- SFT 1M
- CPT 1B
- CPT 5B
- CPT 5B + SFT 1M
- tokenizer swap v2 algorithm
- latest public heldout 2K validation/evaluation set

Trained model weights, optimizer states, generated packed data, and experiment logs are intentionally excluded.

## Repository Structure

```text
dataset_building/        Build SFT/CPT/validation datasets and keep final manifests
model_building/          Build the tokenizer-swapped base model with the v2 algorithm
model_training/          Train SFT/CPT models from existing model and data artifacts
evaluation_reporting/    Run heldout evaluation, merge shards, and generate summaries
```

Each workflow folder has its own README with stage-specific inputs, outputs, scripts, and artifact policy.

The four folders are independent stages:

1. `dataset_building/` produces data artifacts.
2. `model_building/` produces the remapped base checkpoint.
3. `model_training/` consumes a checkpoint plus data and produces trained checkpoints.
4. `evaluation_reporting/` consumes checkpoints plus heldout data and produces reports.

Runtime outputs stay inside these folders but are ignored by git:

```text
dataset_building/generated/
model_building/generated_models/
model_training/checkpoints/
evaluation_reporting/outputs/
evaluation_reporting/reports/
```

## 1. Dataset Building

This folder contains the builders for:

- SFT 1M chat mixture and DSV4-tokenized chat JSONL
- CPT document mixtures
- CPT packed training/eval blocks
- public heldout 2K validation/evaluation set

The only heldout dataset kept in the repo is:

```text
dataset_building/heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
```

The in-domain heldout and earlier ratio-imbalanced 2K heldout are not included.

### SFT Data

The final SFT recipe is the no-upsample, no-BBH v4 chat mix derived from the 1M instruction mixture. Build metadata is kept in:

```text
dataset_building/metadata/sft_v4_mix_build_stats.json
dataset_building/metadata/sft_v4_tokenization_build_stats.json
```

The final training scripts expect generated tokenized SFT data under:

```text
dataset_building/generated/dsv4_chat_tokenized_v4_noupsample_nobbh_921k/
  train_dsv4_chat_tokenized.jsonl.gz
  validation_dsv4_chat_tokenized.jsonl.gz
```

### CPT Data

The final CPT data uses stratified packing with sequence length 8192 and seed 42.

Target source proportions:

| Source bucket | Ratio |
|---|---:|
| English web | 25% |
| English education | 20% |
| Chinese clean | 25% |
| Code | 15% |
| Math | 10% |
| Science | 3% |
| QA as text | 2% |

Final token counts:

| Bucket | CPT 1B source tokens | CPT 5B source tokens |
|---|---:|---:|
| English web | 249,999,374 | 1,249,998,964 |
| English education | 199,998,420 | 999,999,328 |
| Chinese clean | 249,999,618 | 1,249,999,598 |
| Code | 149,999,958 | 749,998,673 |
| Math | 99,987,860 | 499,995,208 |
| Science | 29,999,428 | 149,998,447 |
| QA as text | 19,999,497 | 99,999,341 |

Packed output sizes:

| Dataset | Train tokens | Eval tokens | Train blocks | Eval blocks | Seq len |
|---|---:|---:|---:|---:|---:|
| CPT 1B | 991,567,872 | 8,388,608 | 121,041 | 1,024 | 8192 |
| CPT 5B | 4,983,177,216 | 16,777,216 | 608,298 | 2,048 | 8192 |

Source families:

- FineWeb for English web
- FineWeb-Edu for English education
- BAAI/CCI3-HQ cleaned Chinese data
- StarCoderData/code parquet sources
- OpenWebMath-derived documents
- science reasoning datasets including MedMCQA, ProofWriter, ScienceQA, MedQA, SciQ, QASC, and OpenBookQA
- QA rendered as plain text from the final instruction mixture plus recovery fallback sources

Final manifests are kept in `dataset_building/metadata/`.

## 2. Model Building

`model_building/build_qwen3_dsv4_remap_checkpoint_v2.py` builds the tokenizer-swapped base checkpoint.

Run:

```bash
ROOT=/ssd/yi/Tokenizer_Swap bash model_building/run_remap_v2.sh
```

The algorithm:

- load the source Qwen model/tokenizer and target DSV4 tokenizer
- resize/rebuild input embedding and LM-head rows to the DSV4 vocab size
- initialize each target token row by priority:
  - exact same token surface in Qwen vocab
  - functional special-token mapping, for example DSV BOS to Qwen `<|im_start|>` and DSV EOS to Qwen EOS
  - byte-level decode followed by Qwen tokenization, averaging old rows
  - raw token decomposition fallback, averaging old rows
  - global embedding/head mean fallback
- save the remapped checkpoint and `tokenizer_remap_v2_report.json`

The model-building step does not build datasets and does not train.

## 3. Model Training

Training scripts consume existing model/data paths. They do not perform tokenizer remapping or dataset construction.

Final entrypoints:

| Experiment | Script |
|---|---|
| SFT 1M on remapped base | `model_training/run_sft1m_remap_v2_5epoch.sh` |
| SFT 1M plus v4 no-upsample continuation | `model_training/run_sft1m_remap_v2_then_v4_noupsample_5epoch_bsz16.sh` |
| CPT 1B | `model_training/run_cpt1b_seed42_train_eval.sh` |
| CPT 5B | `model_training/run_cpt5b_seed42_train_eval.sh` |
| CPT 5B + SFT 1M | `model_training/run_cpt5b_then_sft1m_5epoch.sh` |

Most paths are configurable with environment variables:

```bash
MODEL=... DATA=... TRAIN=... EVAL=... OUT=... NPROC=8 bash model_training/run_cpt5b_then_sft1m_5epoch.sh
```

Default outputs go to `model_training/checkpoints/`, which is ignored by git.

## 4. Evaluation And Report Generation

The public heldout 2K evaluation entrypoint is:

```bash
ROOT=/ssd/yi/Tokenizer_Swap \
MODEL=/path/to/checkpoint \
LABEL=my_model \
bash evaluation_reporting/run_public_heldout_eval_8gpu.sh
```

This runs sharded evaluation, merges per-shard outputs, and writes summaries under `evaluation_reporting/outputs/`.

Main public heldout 2K results from the final sweep:

| Model | MCQ acc avg-norm | MCQ acc sum | PPL | NLL/token |
|---|---:|---:|---:|---:|
| Native Qwen3-0.6B tokenizer baseline | 0.2960 | 0.2510 | 61.08 | 3.5399 |
| Remap v2, no training | 0.2940 | 0.2410 | 313.62 | 4.6920 |
| Remap v2 + CPT 1B | 0.3005 | 0.2540 | 71.90 | 3.6580 |
| Remap v2 + CPT 5B | 0.3020 | 0.2615 | 66.95 | 3.5806 |
| Remap v2 + SFT 1M | 0.3105 | 0.2590 | 114.75 | 3.9400 |
| Remap v2 + SFT 1M + v4 continuation | 0.3165 | 0.2595 | 117.33 | 3.9755 |
| Remap v2 + CPT 5B + SFT 1M | 0.3280 | 0.2740 | 88.76 | 3.7899 |

CPT mainly repairs language-modeling quality after tokenizer replacement. SFT improves heldout MCQ/task behavior but can raise perplexity because the objective focuses on assistant-answer tokens rather than generic next-token modeling over heldout text.

## Artifact Policy

Do not commit:

- trained checkpoints or partial checkpoints
- optimizer states
- generated packed CPT data
- generated SFT `.jsonl.gz` data
- `.safetensors`, `.bin`, `.pt`, `.npy`, `.parquet`
- experiment logs and evaluation output directories

The original repo's experiment logs were cleared during migration.