# Model Training

This folder owns full-parameter training recipes for the final experiments.

It does not build datasets and does not perform tokenizer remapping. It consumes artifacts produced by `dataset_building/` and `model_building/`.

## Training Implementations

```text
train_dsv4_tokenized_full_sft.py
train_cpt_packed_full.py
```

`train_dsv4_tokenized_full_sft.py` trains on pre-tokenized chat JSONL. Prompt tokens are masked with `-100`; assistant tokens are optimized.

`train_cpt_packed_full.py` trains next-token prediction over packed CPT blocks.

## Final Recipe Scripts

```text
run_sft1m_remap_v2_5epoch.sh
run_sft1m_remap_v2_then_v4_noupsample_5epoch_bsz16.sh
run_cpt1b_seed42_train_eval.sh
run_cpt5b_seed42_train_eval.sh
run_cpt5b_then_sft1m_5epoch.sh
```

## Common Environment Variables

The run scripts are configurable through environment variables:

```text
ROOT     repo root, default /ssd/yi/Tokenizer_Swap
NPROC    number of GPUs/processes, default 8
MODEL    input checkpoint
DATA     packed CPT dataset directory
TRAIN    tokenized SFT train file
EVAL     tokenized SFT validation file
OUT      output checkpoint directory
```

Example:

```bash
ROOT=/ssd/yi/Tokenizer_Swap \
MODEL=/ssd/yi/Tokenizer_Swap/model_building/generated_models/Qwen3-0.6B-DSV4-tokenizer-remap-v2 \
DATA=/ssd/yi/Tokenizer_Swap/dataset_building/generated/cpt_packed_5b_seq8192_seed42_stratified \
OUT=/ssd/yi/Tokenizer_Swap/model_training/checkpoints/cpt5b \
bash model_training/run_cpt5b_seed42_train_eval.sh
```

## Output

Generated checkpoints go under:

```text
model_training/checkpoints/
```

This directory is ignored by git. Do not commit model weights, optimizer states, or partial checkpoints.

## Output Contract

Trained checkpoints are consumed by `evaluation_reporting/` as `MODEL`.