Tokenizer_Swap/README.md

Tokenizer Swap

This is the cleaned migration of the tokenizer-swap experiments. The repo is organized by workflow boundary, not by experiment date or temporary artifact type.

It keeps only the final useful recipes:

• SFT 1M
• CPT 1B
• CPT 5B
• CPT 5B + SFT 1M
• tokenizer swap v2 algorithm
• latest public heldout 2K validation/evaluation set

Trained model weights, optimizer states, generated packed data, and experiment logs are intentionally excluded.

Repository Structure

dataset_building/        Build SFT/CPT/validation datasets and keep final manifests
model_building/          Build the tokenizer-swapped base model with the v2 algorithm
model_training/          Train SFT/CPT models from existing model and data artifacts
evaluation_reporting/    Run heldout evaluation, merge shards, and generate summaries

Each workflow folder has its own README with stage-specific inputs, outputs, scripts, and artifact policy.

The four folders are independent stages:

1. dataset_building/ produces data artifacts.
2. model_building/ produces the remapped base checkpoint.
3. model_training/ consumes a checkpoint plus data and produces trained checkpoints.
4. evaluation_reporting/ consumes checkpoints plus heldout data and produces reports.

Runtime outputs stay inside these folders but are ignored by git:

dataset_building/generated/
model_building/generated_models/
model_training/checkpoints/
evaluation_reporting/outputs/
evaluation_reporting/reports/

1. Dataset Building

This folder contains the builders for:

• SFT 1M chat mixture and DSV4-tokenized chat JSONL
• CPT document mixtures
• CPT packed training/eval blocks
• public heldout 2K validation/evaluation set

The only heldout dataset kept in the repo is:

dataset_building/heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl

The in-domain heldout and earlier ratio-imbalanced 2K heldout are not included.

SFT Data

The final SFT recipe is the no-upsample, no-BBH v4 chat mix derived from the 1M instruction mixture. Build metadata is kept in:

dataset_building/metadata/sft_v4_mix_build_stats.json
dataset_building/metadata/sft_v4_tokenization_build_stats.json

The final training scripts expect generated tokenized SFT data under:

dataset_building/generated/dsv4_chat_tokenized_v4_noupsample_nobbh_921k/
  train_dsv4_chat_tokenized.jsonl.gz
  validation_dsv4_chat_tokenized.jsonl.gz

CPT Data

The final CPT data uses stratified packing with sequence length 8192 and seed 42.

Target source proportions:

Source bucket	Ratio
English web	25%
English education	20%
Chinese clean	25%
Code	15%
Math	10%
Science	3%
QA as text	2%

Final token counts:

Bucket	CPT 1B source tokens	CPT 5B source tokens
English web	249,999,374	1,249,998,964
English education	199,998,420	999,999,328
Chinese clean	249,999,618	1,249,999,598
Code	149,999,958	749,998,673
Math	99,987,860	499,995,208
Science	29,999,428	149,998,447
QA as text	19,999,497	99,999,341

Packed output sizes:

Dataset	Train tokens	Eval tokens	Train blocks	Eval blocks	Seq len
CPT 1B	991,567,872	8,388,608	121,041	1,024	8192
CPT 5B	4,983,177,216	16,777,216	608,298	2,048	8192

Source families:

• FineWeb for English web
• FineWeb-Edu for English education
• BAAI/CCI3-HQ cleaned Chinese data
• StarCoderData/code parquet sources
• OpenWebMath-derived documents
• science reasoning datasets including MedMCQA, ProofWriter, ScienceQA, MedQA, SciQ, QASC, and OpenBookQA
• QA rendered as plain text from the final instruction mixture plus recovery fallback sources

Final manifests are kept in dataset_building/metadata/.

2. Model Building

model_building/build_qwen3_dsv4_remap_checkpoint_v2.py builds the tokenizer-swapped base checkpoint.

Run:

ROOT=/ssd/yi/Tokenizer_Swap bash model_building/run_remap_v2.sh

The algorithm:

• load the source Qwen model/tokenizer and target DSV4 tokenizer
• resize/rebuild input embedding and LM-head rows to the DSV4 vocab size
• initialize each target token row by priority:
  • exact same token surface in Qwen vocab
  • functional special-token mapping, for example DSV BOS to Qwen <|im_start|> and DSV EOS to Qwen EOS
  • byte-level decode followed by Qwen tokenization, averaging old rows
  • raw token decomposition fallback, averaging old rows
  • global embedding/head mean fallback
• save the remapped checkpoint and tokenizer_remap_v2_report.json

The model-building step does not build datasets and does not train.

3. Model Training

Training scripts consume existing model/data paths. They do not perform tokenizer remapping or dataset construction.

Final entrypoints:

Experiment	Script
SFT 1M on remapped base	model_training/run_sft1m_remap_v2_5epoch.sh
SFT 1M plus v4 no-upsample continuation	model_training/run_sft1m_remap_v2_then_v4_noupsample_5epoch_bsz16.sh
CPT 1B	model_training/run_cpt1b_seed42_train_eval.sh
CPT 5B	model_training/run_cpt5b_seed42_train_eval.sh
CPT 5B + SFT 1M	model_training/run_cpt5b_then_sft1m_5epoch.sh

Most paths are configurable with environment variables:

MODEL=... DATA=... TRAIN=... EVAL=... OUT=... NPROC=8 bash model_training/run_cpt5b_then_sft1m_5epoch.sh

Default outputs go to model_training/checkpoints/, which is ignored by git.

4. Evaluation And Report Generation

The public heldout 2K evaluation entrypoint is:

ROOT=/ssd/yi/Tokenizer_Swap \
MODEL=/path/to/checkpoint \
LABEL=my_model \
bash evaluation_reporting/run_public_heldout_eval_8gpu.sh

This runs sharded evaluation, merges per-shard outputs, and writes summaries under evaluation_reporting/outputs/.

Main public heldout 2K results from the final sweep:

Model	MCQ acc avg-norm	MCQ acc sum	PPL	NLL/token
Native Qwen3-0.6B tokenizer baseline	0.2960	0.2510	61.08	3.5399
Remap v2, no training	0.2940	0.2410	313.62	4.6920
Remap v2 + CPT 1B	0.3005	0.2540	71.90	3.6580
Remap v2 + CPT 5B	0.3020	0.2615	66.95	3.5806
Remap v2 + SFT 1M	0.3105	0.2590	114.75	3.9400
Remap v2 + SFT 1M + v4 continuation	0.3165	0.2595	117.33	3.9755
Remap v2 + CPT 5B + SFT 1M	0.3280	0.2740	88.76	3.7899

CPT mainly repairs language-modeling quality after tokenizer replacement. SFT improves heldout MCQ/task behavior but can raise perplexity because the objective focuses on assistant-answer tokens rather than generic next-token modeling over heldout text.

Artifact Policy

Do not commit:

• trained checkpoints or partial checkpoints
• optimizer states
• generated packed CPT data
• generated SFT .jsonl.gz data
• .safetensors, .bin, .pt, .npy, .parquet
• experiment logs and evaluation output directories

The original repo's experiment logs were cleared during migration.