Dataset Building
This folder owns all dataset construction for the migration: SFT data, CPT data, and the public heldout validation/evaluation set.
It does not build model weights and does not run training.
Inputs
The builders expect raw or intermediate source data outside the git repo, such as FineWeb/FineWeb-Edu shards, BAAI/CCI3-HQ Chinese data, StarCoder/code data, OpenWebMath-derived data, science reasoning caches, and instruction QA mixes.
Tokenizer paths are configurable with CLI flags. Generated data should be written under:
dataset_building/generated/
That directory is ignored by git.
Kept Dataset
Only the latest public heldout 2K set is versioned here:
heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
heldout_public_mcq_2k_20260607/heldout_public_mcq_2k_stats.json
Older in-domain heldout and earlier ratio-imbalanced 2K heldout datasets are intentionally not included.
SFT Builders
Main scripts:
build_training_and_test_mix_v3.py
build_dsv4_chat_tokenized_messages_jsonl.py
build_dsv4_chat_tokenized_custom.py
Expected final generated layout:
dataset_building/generated/dsv4_chat_tokenized_v4_noupsample_nobbh_921k/
train_dsv4_chat_tokenized.jsonl.gz
validation_dsv4_chat_tokenized.jsonl.gz
Build metadata from the final SFT recipe is kept in:
metadata/sft_v4_mix_build_stats.json
metadata/sft_v4_tokenization_build_stats.json
CPT Builders
Main scripts:
build_cpt_docmix_1b.py
build_cpt_packed_stratified.py
build_cpt_packed_5b_stratified.py
build_cci3_chinese_docmix_fix.py
build_math_docmix_fix.py
build_science_docmix_fix.py
The final CPT recipes use stratified packing with sequence length 8192 and seed 42.
Source proportions:
| Source bucket | Ratio |
|---|---|
| English web | 25% |
| English education | 20% |
| Chinese clean | 25% |
| Code | 15% |
| Math | 10% |
| Science | 3% |
| QA as text | 2% |
Final manifests are kept in:
metadata/cpt_packed_1b_seed42_stratified_manifest.json
metadata/cpt_packed_5b_seed42_stratified_manifest.json
metadata/cpt_docmix_5b_manifest.json
metadata/cpt_docmix_5b_stats.json
Output Contract
Downstream training expects either:
- tokenized SFT
.jsonl.gzfiles fortrain_dsv4_tokenized_full_sft.py - packed CPT arrays/manifests for
train_cpt_packed_full.py
Do not commit generated .jsonl.gz, .npy, .parquet, or other large intermediate files.