Clean tokenizer swap migration

2026-06-18 10:10:57 +00:00
commit 5d1e9c4bc3
37 changed files with 9574 additions and 0 deletions
@@ -0,0 +1,98 @@
+# Dataset Building
+
+This folder owns all dataset construction for the migration: SFT data, CPT data, and the public heldout validation/evaluation set.
+
+It does not build model weights and does not run training.
+
+## Inputs
+
+The builders expect raw or intermediate source data outside the git repo, such as FineWeb/FineWeb-Edu shards, BAAI/CCI3-HQ Chinese data, StarCoder/code data, OpenWebMath-derived data, science reasoning caches, and instruction QA mixes.
+
+Tokenizer paths are configurable with CLI flags. Generated data should be written under:
+
+```text
+dataset_building/generated/
+```
+
+That directory is ignored by git.
+
+## Kept Dataset
+
+Only the latest public heldout 2K set is versioned here:
+
+```text
+heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
+heldout_public_mcq_2k_20260607/heldout_public_mcq_2k_stats.json
+```
+
+Older in-domain heldout and earlier ratio-imbalanced 2K heldout datasets are intentionally not included.
+
+## SFT Builders
+
+Main scripts:
+
+```text
+build_training_and_test_mix_v3.py
+build_dsv4_chat_tokenized_messages_jsonl.py
+build_dsv4_chat_tokenized_custom.py
+```
+
+Expected final generated layout:
+
+```text
+dataset_building/generated/dsv4_chat_tokenized_v4_noupsample_nobbh_921k/
+  train_dsv4_chat_tokenized.jsonl.gz
+  validation_dsv4_chat_tokenized.jsonl.gz
+```
+
+Build metadata from the final SFT recipe is kept in:
+
+```text
+metadata/sft_v4_mix_build_stats.json
+metadata/sft_v4_tokenization_build_stats.json
+```
+
+## CPT Builders
+
+Main scripts:
+
+```text
+build_cpt_docmix_1b.py
+build_cpt_packed_stratified.py
+build_cpt_packed_5b_stratified.py
+build_cci3_chinese_docmix_fix.py
+build_math_docmix_fix.py
+build_science_docmix_fix.py
+```
+
+The final CPT recipes use stratified packing with sequence length 8192 and seed 42.
+
+Source proportions:
+
+| Source bucket | Ratio |
+|---|---:|
+| English web | 25% |
+| English education | 20% |
+| Chinese clean | 25% |
+| Code | 15% |
+| Math | 10% |
+| Science | 3% |
+| QA as text | 2% |
+
+Final manifests are kept in:
+
+```text
+metadata/cpt_packed_1b_seed42_stratified_manifest.json
+metadata/cpt_packed_5b_seed42_stratified_manifest.json
+metadata/cpt_docmix_5b_manifest.json
+metadata/cpt_docmix_5b_stats.json
+```
+
+## Output Contract
+
+Downstream training expects either:
+
+- tokenized SFT `.jsonl.gz` files for `train_dsv4_tokenized_full_sft.py`
+- packed CPT arrays/manifests for `train_cpt_packed_full.py`
+
+Do not commit generated `.jsonl.gz`, `.npy`, `.parquet`, or other large intermediate files.