Add proof-pile logic inputs to pretrain builder

This commit is contained in:
2026-07-01 22:42:18 +08:00
parent cdd8e2f85b
commit f0f72767f5
11 changed files with 122 additions and 54 deletions

View File

@@ -44,16 +44,10 @@ repo 路径:
/mnt/beegfs/yi/laoyao_2b_moe
```
200B 数据当前构建源路径:
预训练数据实际读取路径:
```bash
/mnt/beegfs/yi/laoyao_2b_moe_pretraining_dataset/train/pretrain_rebalanced_web40_edu20_chinese10_science10_logic10_math5_code5_200b_v1_20260701
```
构建完成后同步到 repo 内
```bash
bash scripts/sync_pretrain_data_into_repo.sh
```
同步后的数据目录 `dataset/pretrain/data/``.gitignore` 忽略,不提交到 GitHub。
数据不再复制到 repo 内;训练直接读取上面的源输出目录。`dataset/pretrain/data/` 仍被 `.gitignore` 忽略,仅用于小规模临时样本。