65 lines
1.8 KiB
Markdown
65 lines
1.8 KiB
Markdown
# Model Training
|
|
|
|
This folder owns full-parameter training recipes for the final experiments.
|
|
|
|
It does not build datasets and does not perform tokenizer remapping. It consumes artifacts produced by `dataset_building/` and `model_building/`.
|
|
|
|
## Training Implementations
|
|
|
|
```text
|
|
train_dsv4_tokenized_full_sft.py
|
|
train_cpt_packed_full.py
|
|
```
|
|
|
|
`train_dsv4_tokenized_full_sft.py` trains on pre-tokenized chat JSONL. Prompt tokens are masked with `-100`; assistant tokens are optimized.
|
|
|
|
`train_cpt_packed_full.py` trains next-token prediction over packed CPT blocks.
|
|
|
|
## Final Recipe Scripts
|
|
|
|
```text
|
|
run_sft1m_remap_v2_5epoch.sh
|
|
run_sft1m_remap_v2_then_v4_noupsample_5epoch_bsz16.sh
|
|
run_cpt1b_seed42_train_eval.sh
|
|
run_cpt5b_seed42_train_eval.sh
|
|
run_cpt5b_then_sft1m_5epoch.sh
|
|
```
|
|
|
|
## Common Environment Variables
|
|
|
|
The run scripts are configurable through environment variables:
|
|
|
|
```text
|
|
ROOT repo root, default /ssd/yi/Tokenizer_Swap
|
|
NPROC number of GPUs/processes, default 8
|
|
MODEL input checkpoint
|
|
DATA packed CPT dataset directory
|
|
TRAIN tokenized SFT train file
|
|
EVAL tokenized SFT validation file
|
|
OUT output checkpoint directory
|
|
```
|
|
|
|
Example:
|
|
|
|
```bash
|
|
ROOT=/ssd/yi/Tokenizer_Swap \
|
|
MODEL=/ssd/yi/Tokenizer_Swap/model_building/generated_models/Qwen3-0.6B-DSV4-tokenizer-remap-v2 \
|
|
DATA=/ssd/yi/Tokenizer_Swap/dataset_building/generated/cpt_packed_5b_seq8192_seed42_stratified \
|
|
OUT=/ssd/yi/Tokenizer_Swap/model_training/checkpoints/cpt5b \
|
|
bash model_training/run_cpt5b_seed42_train_eval.sh
|
|
```
|
|
|
|
## Output
|
|
|
|
Generated checkpoints go under:
|
|
|
|
```text
|
|
model_training/checkpoints/
|
|
```
|
|
|
|
This directory is ignored by git. Do not commit model weights, optimizer states, or partial checkpoints.
|
|
|
|
## Output Contract
|
|
|
|
Trained checkpoints are consumed by `evaluation_reporting/` as `MODEL`.
|