Dataset Building

This folder owns all dataset construction for the migration: SFT data, CPT data, and the public heldout validation/evaluation set.

It does not build model weights and does not run training.

Inputs

The builders expect raw or intermediate source data outside the git repo, such as FineWeb/FineWeb-Edu shards, BAAI/CCI3-HQ Chinese data, StarCoder/code data, OpenWebMath-derived data, science reasoning caches, and instruction QA mixes.

Tokenizer paths are configurable with CLI flags. Generated data should be written under:

dataset_building/generated/

That directory is ignored by git.

Kept Datasets

The latest public heldout 2K set is versioned here:

heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
heldout_public_mcq_2k_20260607/heldout_public_mcq_2k_stats.json

The 2.8K in-domain SFT heldout is kept as a reproducible build recipe rather than checked-in generated data. It contains seven ability dimensions with 400 examples each:

science_reasoning
logic
code
chinese_exam
math
chinese_dialogue
english_dialogue

Build it with:

bash dataset_building/build_sft_heldout_2p8k_20260611.sh

By default this writes:

dataset_building/generated/training_mix_v4_train1m_test2p8k_noupsample_nobbh_20260611/
  heldout_2p8k.jsonl
  heldout_exclusion_hashes.jsonl
  train_1m.jsonl
  build_stats.json

Use OUT=/path/to/output and TOKENIZER=/path/to/tokenizer to place the validation set outside the repo or build it with a specific tokenizer. For Laoyao custom BPE tokenizers, set LAOYAO_MODEL_ROOT=/path/to/laoyao/repo so the builder can import laoyaomodel.tokenization.bpe. Generated outputs stay ignored by git.

SFT Builders

Main scripts:

build_training_and_test_mix_v3.py
build_dsv4_chat_tokenized_messages_jsonl.py
build_dsv4_chat_tokenized_custom.py

Expected final generated layout:

dataset_building/generated/dsv4_chat_tokenized_v4_noupsample_nobbh_921k/
  train_dsv4_chat_tokenized.jsonl.gz
  validation_dsv4_chat_tokenized.jsonl.gz

Build metadata from the final SFT recipe is kept in:

metadata/sft_v4_mix_build_stats.json
metadata/sft_v4_tokenization_build_stats.json

CPT Builders

Main scripts:

build_cpt_docmix_1b.py
build_cpt_packed_stratified.py
build_cpt_packed_5b_stratified.py
build_cci3_chinese_docmix_fix.py
build_math_docmix_fix.py
build_science_docmix_fix.py

The final CPT recipes use stratified packing with sequence length 8192 and seed 42.

Source proportions:

Source bucket	Ratio
English web	25%
English education	20%
Chinese clean	25%
Code	15%
Math	10%
Science	3%
QA as text	2%

Final manifests are kept in:

metadata/cpt_packed_1b_seed42_stratified_manifest.json
metadata/cpt_packed_5b_seed42_stratified_manifest.json
metadata/cpt_docmix_5b_manifest.json
metadata/cpt_docmix_5b_stats.json

Output Contract

Downstream training expects either:

tokenized SFT .jsonl.gz files for train_dsv4_tokenized_full_sft.py
packed CPT arrays/manifests for train_cpt_packed_full.py

Do not commit generated .jsonl.gz, .npy, .parquet, or other large intermediate files.

3.2 KiB Raw Permalink Blame History