Add proof-pile logic inputs to pretrain builder
This commit is contained in:
10
README.md
10
README.md
@@ -44,16 +44,10 @@ repo 路径:
|
||||
/mnt/beegfs/yi/laoyao_2b_moe
|
||||
```
|
||||
|
||||
200B 数据当前构建源路径:
|
||||
预训练数据实际读取路径:
|
||||
|
||||
```bash
|
||||
/mnt/beegfs/yi/laoyao_2b_moe_pretraining_dataset/train/pretrain_rebalanced_web40_edu20_chinese10_science10_logic10_math5_code5_200b_v1_20260701
|
||||
```
|
||||
|
||||
构建完成后同步到 repo 内:
|
||||
|
||||
```bash
|
||||
bash scripts/sync_pretrain_data_into_repo.sh
|
||||
```
|
||||
|
||||
同步后的数据目录 `dataset/pretrain/data/` 被 `.gitignore` 忽略,不提交到 GitHub。
|
||||
数据不再复制到 repo 内;训练直接读取上面的源输出目录。`dataset/pretrain/data/` 仍被 `.gitignore` 忽略,仅用于小规模临时样本。
|
||||
|
||||
Reference in New Issue
Block a user