133 lines
3.2 KiB
Markdown
133 lines
3.2 KiB
Markdown
# Dataset Building
|
|
|
|
This folder owns all dataset construction for the migration: SFT data, CPT data, and the public heldout validation/evaluation set.
|
|
|
|
It does not build model weights and does not run training.
|
|
|
|
## Inputs
|
|
|
|
The builders expect raw or intermediate source data outside the git repo, such as FineWeb/FineWeb-Edu shards, BAAI/CCI3-HQ Chinese data, StarCoder/code data, OpenWebMath-derived data, science reasoning caches, and instruction QA mixes.
|
|
|
|
Tokenizer paths are configurable with CLI flags. Generated data should be written under:
|
|
|
|
```text
|
|
dataset_building/generated/
|
|
```
|
|
|
|
That directory is ignored by git.
|
|
|
|
## Kept Datasets
|
|
|
|
The latest public heldout 2K set is versioned here:
|
|
|
|
```text
|
|
heldout_public_mcq_2k_20260607/heldout_public_mcq_2k.jsonl
|
|
heldout_public_mcq_2k_20260607/heldout_public_mcq_2k_stats.json
|
|
```
|
|
|
|
The 2.8K in-domain SFT heldout is kept as a reproducible build recipe rather
|
|
than checked-in generated data. It contains seven ability dimensions with 400
|
|
examples each:
|
|
|
|
```text
|
|
science_reasoning
|
|
logic
|
|
code
|
|
chinese_exam
|
|
math
|
|
chinese_dialogue
|
|
english_dialogue
|
|
```
|
|
|
|
Build it with:
|
|
|
|
```bash
|
|
bash dataset_building/build_sft_heldout_2p8k_20260611.sh
|
|
```
|
|
|
|
By default this writes:
|
|
|
|
```text
|
|
dataset_building/generated/training_mix_v4_train1m_test2p8k_noupsample_nobbh_20260611/
|
|
heldout_2p8k.jsonl
|
|
heldout_exclusion_hashes.jsonl
|
|
train_1m.jsonl
|
|
build_stats.json
|
|
```
|
|
|
|
Use `OUT=/path/to/output` and `TOKENIZER=/path/to/tokenizer` to place the
|
|
validation set outside the repo or build it with a specific tokenizer. For
|
|
Laoyao custom BPE tokenizers, set `LAOYAO_MODEL_ROOT=/path/to/laoyao/repo` so
|
|
the builder can import `laoyaomodel.tokenization.bpe`. Generated outputs stay
|
|
ignored by git.
|
|
|
|
## SFT Builders
|
|
|
|
Main scripts:
|
|
|
|
```text
|
|
build_training_and_test_mix_v3.py
|
|
build_dsv4_chat_tokenized_messages_jsonl.py
|
|
build_dsv4_chat_tokenized_custom.py
|
|
```
|
|
|
|
Expected final generated layout:
|
|
|
|
```text
|
|
dataset_building/generated/dsv4_chat_tokenized_v4_noupsample_nobbh_921k/
|
|
train_dsv4_chat_tokenized.jsonl.gz
|
|
validation_dsv4_chat_tokenized.jsonl.gz
|
|
```
|
|
|
|
Build metadata from the final SFT recipe is kept in:
|
|
|
|
```text
|
|
metadata/sft_v4_mix_build_stats.json
|
|
metadata/sft_v4_tokenization_build_stats.json
|
|
```
|
|
|
|
## CPT Builders
|
|
|
|
Main scripts:
|
|
|
|
```text
|
|
build_cpt_docmix_1b.py
|
|
build_cpt_packed_stratified.py
|
|
build_cpt_packed_5b_stratified.py
|
|
build_cci3_chinese_docmix_fix.py
|
|
build_math_docmix_fix.py
|
|
build_science_docmix_fix.py
|
|
```
|
|
|
|
The final CPT recipes use stratified packing with sequence length 8192 and seed 42.
|
|
|
|
Source proportions:
|
|
|
|
| Source bucket | Ratio |
|
|
|---|---:|
|
|
| English web | 25% |
|
|
| English education | 20% |
|
|
| Chinese clean | 25% |
|
|
| Code | 15% |
|
|
| Math | 10% |
|
|
| Science | 3% |
|
|
| QA as text | 2% |
|
|
|
|
Final manifests are kept in:
|
|
|
|
```text
|
|
metadata/cpt_packed_1b_seed42_stratified_manifest.json
|
|
metadata/cpt_packed_5b_seed42_stratified_manifest.json
|
|
metadata/cpt_docmix_5b_manifest.json
|
|
metadata/cpt_docmix_5b_stats.json
|
|
```
|
|
|
|
## Output Contract
|
|
|
|
Downstream training expects either:
|
|
|
|
- tokenized SFT `.jsonl.gz` files for `train_dsv4_tokenized_full_sft.py`
|
|
- packed CPT arrays/manifests for `train_cpt_packed_full.py`
|
|
|
|
Do not commit generated `.jsonl.gz`, `.npy`, `.parquet`, or other large intermediate files.
|