210 lines
6.9 KiB
Markdown
210 lines
6.9 KiB
Markdown
# Tokenizer Swap
|
|
|
|
This is the cleaned migration of the tokenizer-swap experiments. The repo is organized by workflow boundary, not by experiment date or temporary artifact type.
|
|
|
|
It keeps only the final useful recipes:
|
|
|
|
- SFT 1M
|
|
- CPT 1B
|
|
- CPT 5B
|
|
- CPT 5B + SFT 1M
|
|
- tokenizer swap v2 algorithm
|
|
- latest public heldout 2K validation/evaluation set
|
|
|
|
Trained model weights, optimizer states, generated packed data, and experiment logs are intentionally excluded.
|
|
|
|
## Repository Structure
|
|
|
|
```text
|
|
dataset_building/ Build SFT/CPT/validation datasets and keep final manifests
|
|
model_building/ Build the tokenizer-swapped base model with the v2 algorithm
|
|
model_training/ Train SFT/CPT models from existing model and data artifacts
|
|
evaluation_reporting/ Run heldout evaluation, merge shards, and generate summaries
|
|
```
|
|
|
|
Each workflow folder has its own README with stage-specific inputs, outputs, scripts, and artifact policy.
|
|
|
|
The four folders are independent stages:
|
|
|
|
1. `dataset_building/` produces data artifacts.
|
|
2. `model_building/` produces the remapped base checkpoint.
|
|
3. `model_training/` consumes a checkpoint plus data and produces trained checkpoints.
|
|
4. `evaluation_reporting/` consumes checkpoints plus heldout data and produces reports.
|
|
|
|
Runtime outputs stay inside these folders but are ignored by git:
|
|
|
|
```text
|
|
dataset_building/generated/
|
|
model_building/generated_models/
|
|
model_training/checkpoints/
|
|
evaluation_reporting/outputs/
|
|
evaluation_reporting/reports/
|
|
```
|
|
|
|
## 1. Dataset Building
|
|
|
|
This folder contains the builders for:
|
|
|
|
- SFT 1M chat mixture and DSV4-tokenized chat JSONL
|
|
- CPT document mixtures
|
|
- CPT packed training/eval blocks
|
|
- public heldout 2K validation/evaluation set
|
|
|
|
The only heldout dataset kept in the repo is:
|
|
|
|
```text
|
|
dataset_building/heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
|
|
```
|
|
|
|
The in-domain heldout and earlier ratio-imbalanced 2K heldout are not included.
|
|
|
|
### SFT Data
|
|
|
|
The final SFT recipe is the no-upsample, no-BBH v4 chat mix derived from the 1M instruction mixture. Build metadata is kept in:
|
|
|
|
```text
|
|
dataset_building/metadata/sft_v4_mix_build_stats.json
|
|
dataset_building/metadata/sft_v4_tokenization_build_stats.json
|
|
```
|
|
|
|
The final training scripts expect generated tokenized SFT data under:
|
|
|
|
```text
|
|
dataset_building/generated/dsv4_chat_tokenized_v4_noupsample_nobbh_921k/
|
|
train_dsv4_chat_tokenized.jsonl.gz
|
|
validation_dsv4_chat_tokenized.jsonl.gz
|
|
```
|
|
|
|
### CPT Data
|
|
|
|
The final CPT data uses stratified packing with sequence length 8192 and seed 42.
|
|
|
|
Target source proportions:
|
|
|
|
| Source bucket | Ratio |
|
|
|---|---:|
|
|
| English web | 25% |
|
|
| English education | 20% |
|
|
| Chinese clean | 25% |
|
|
| Code | 15% |
|
|
| Math | 10% |
|
|
| Science | 3% |
|
|
| QA as text | 2% |
|
|
|
|
Final token counts:
|
|
|
|
| Bucket | CPT 1B source tokens | CPT 5B source tokens |
|
|
|---|---:|---:|
|
|
| English web | 249,999,374 | 1,249,998,964 |
|
|
| English education | 199,998,420 | 999,999,328 |
|
|
| Chinese clean | 249,999,618 | 1,249,999,598 |
|
|
| Code | 149,999,958 | 749,998,673 |
|
|
| Math | 99,987,860 | 499,995,208 |
|
|
| Science | 29,999,428 | 149,998,447 |
|
|
| QA as text | 19,999,497 | 99,999,341 |
|
|
|
|
Packed output sizes:
|
|
|
|
| Dataset | Train tokens | Eval tokens | Train blocks | Eval blocks | Seq len |
|
|
|---|---:|---:|---:|---:|---:|
|
|
| CPT 1B | 991,567,872 | 8,388,608 | 121,041 | 1,024 | 8192 |
|
|
| CPT 5B | 4,983,177,216 | 16,777,216 | 608,298 | 2,048 | 8192 |
|
|
|
|
Source families:
|
|
|
|
- FineWeb for English web
|
|
- FineWeb-Edu for English education
|
|
- BAAI/CCI3-HQ cleaned Chinese data
|
|
- StarCoderData/code parquet sources
|
|
- OpenWebMath-derived documents
|
|
- science reasoning datasets including MedMCQA, ProofWriter, ScienceQA, MedQA, SciQ, QASC, and OpenBookQA
|
|
- QA rendered as plain text from the final instruction mixture plus recovery fallback sources
|
|
|
|
Final manifests are kept in `dataset_building/metadata/`.
|
|
|
|
## 2. Model Building
|
|
|
|
`model_building/build_qwen3_dsv4_remap_checkpoint_v2.py` builds the tokenizer-swapped base checkpoint.
|
|
|
|
Run:
|
|
|
|
```bash
|
|
ROOT=/ssd/yi/Tokenizer_Swap bash model_building/run_remap_v2.sh
|
|
```
|
|
|
|
The algorithm:
|
|
|
|
- load the source Qwen model/tokenizer and target DSV4 tokenizer
|
|
- resize/rebuild input embedding and LM-head rows to the DSV4 vocab size
|
|
- initialize each target token row by priority:
|
|
- exact same token surface in Qwen vocab
|
|
- functional special-token mapping, for example DSV BOS to Qwen `<|im_start|>` and DSV EOS to Qwen EOS
|
|
- byte-level decode followed by Qwen tokenization, averaging old rows
|
|
- raw token decomposition fallback, averaging old rows
|
|
- global embedding/head mean fallback
|
|
- save the remapped checkpoint and `tokenizer_remap_v2_report.json`
|
|
|
|
The model-building step does not build datasets and does not train.
|
|
|
|
## 3. Model Training
|
|
|
|
Training scripts consume existing model/data paths. They do not perform tokenizer remapping or dataset construction.
|
|
|
|
Final entrypoints:
|
|
|
|
| Experiment | Script |
|
|
|---|---|
|
|
| SFT 1M on remapped base | `model_training/run_sft1m_remap_v2_5epoch.sh` |
|
|
| SFT 1M plus v4 no-upsample continuation | `model_training/run_sft1m_remap_v2_then_v4_noupsample_5epoch_bsz16.sh` |
|
|
| CPT 1B | `model_training/run_cpt1b_seed42_train_eval.sh` |
|
|
| CPT 5B | `model_training/run_cpt5b_seed42_train_eval.sh` |
|
|
| CPT 5B + SFT 1M | `model_training/run_cpt5b_then_sft1m_5epoch.sh` |
|
|
|
|
Most paths are configurable with environment variables:
|
|
|
|
```bash
|
|
MODEL=... DATA=... TRAIN=... EVAL=... OUT=... NPROC=8 bash model_training/run_cpt5b_then_sft1m_5epoch.sh
|
|
```
|
|
|
|
Default outputs go to `model_training/checkpoints/`, which is ignored by git.
|
|
|
|
## 4. Evaluation And Report Generation
|
|
|
|
The public heldout 2K evaluation entrypoint is:
|
|
|
|
```bash
|
|
ROOT=/ssd/yi/Tokenizer_Swap \
|
|
MODEL=/path/to/checkpoint \
|
|
LABEL=my_model \
|
|
bash evaluation_reporting/run_public_heldout_eval_8gpu.sh
|
|
```
|
|
|
|
This runs sharded evaluation, merges per-shard outputs, and writes summaries under `evaluation_reporting/outputs/`.
|
|
|
|
Main public heldout 2K results from the final sweep:
|
|
|
|
| Model | MCQ acc avg-norm | MCQ acc sum | PPL | NLL/token |
|
|
|---|---:|---:|---:|---:|
|
|
| Native Qwen3-0.6B tokenizer baseline | 0.2960 | 0.2510 | 61.08 | 3.5399 |
|
|
| Remap v2, no training | 0.2940 | 0.2410 | 313.62 | 4.6920 |
|
|
| Remap v2 + CPT 1B | 0.3005 | 0.2540 | 71.90 | 3.6580 |
|
|
| Remap v2 + CPT 5B | 0.3020 | 0.2615 | 66.95 | 3.5806 |
|
|
| Remap v2 + SFT 1M | 0.3105 | 0.2590 | 114.75 | 3.9400 |
|
|
| Remap v2 + SFT 1M + v4 continuation | 0.3165 | 0.2595 | 117.33 | 3.9755 |
|
|
| Remap v2 + CPT 5B + SFT 1M | 0.3280 | 0.2740 | 88.76 | 3.7899 |
|
|
|
|
CPT mainly repairs language-modeling quality after tokenizer replacement. SFT improves heldout MCQ/task behavior but can raise perplexity because the objective focuses on assistant-answer tokens rather than generic next-token modeling over heldout text.
|
|
|
|
## Artifact Policy
|
|
|
|
Do not commit:
|
|
|
|
- trained checkpoints or partial checkpoints
|
|
- optimizer states
|
|
- generated packed CPT data
|
|
- generated SFT `.jsonl.gz` data
|
|
- `.safetensors`, `.bin`, `.pt`, `.npy`, `.parquet`
|
|
- experiment logs and evaluation output directories
|
|
|
|
The original repo's experiment logs were cleared during migration.
|