Clean tokenizer swap migration

model_training/README.md

@@ -0,0 +1,64 @@
+# Model Training
+
+This folder owns full-parameter training recipes for the final experiments.
+
+It does not build datasets and does not perform tokenizer remapping. It consumes artifacts produced by `dataset_building/` and `model_building/`.
+
+## Training Implementations
+
+```text
+train_dsv4_tokenized_full_sft.py
+train_cpt_packed_full.py
+```
+
+`train_dsv4_tokenized_full_sft.py` trains on pre-tokenized chat JSONL. Prompt tokens are masked with `-100`; assistant tokens are optimized.
+
+`train_cpt_packed_full.py` trains next-token prediction over packed CPT blocks.
+
+## Final Recipe Scripts
+
+```text
+run_sft1m_remap_v2_5epoch.sh
+run_sft1m_remap_v2_then_v4_noupsample_5epoch_bsz16.sh
+run_cpt1b_seed42_train_eval.sh
+run_cpt5b_seed42_train_eval.sh
+run_cpt5b_then_sft1m_5epoch.sh
+```
+
+## Common Environment Variables
+
+The run scripts are configurable through environment variables:
+
+```text
+ROOT     repo root, default /ssd/yi/Tokenizer_Swap
+NPROC    number of GPUs/processes, default 8
+MODEL    input checkpoint
+DATA     packed CPT dataset directory
+TRAIN    tokenized SFT train file
+EVAL     tokenized SFT validation file
+OUT      output checkpoint directory
+```
+
+Example:
+
+```bash
+ROOT=/ssd/yi/Tokenizer_Swap \
+MODEL=/ssd/yi/Tokenizer_Swap/model_building/generated_models/Qwen3-0.6B-DSV4-tokenizer-remap-v2 \
+DATA=/ssd/yi/Tokenizer_Swap/dataset_building/generated/cpt_packed_5b_seq8192_seed42_stratified \
+OUT=/ssd/yi/Tokenizer_Swap/model_training/checkpoints/cpt5b \
+bash model_training/run_cpt5b_seed42_train_eval.sh
+```
+
+## Output
+
+Generated checkpoints go under:
+
+```text
+model_training/checkpoints/
+```
+
+This directory is ignored by git. Do not commit model weights, optimizer states, or partial checkpoints.
+
+## Output Contract
+
+Trained checkpoints are consumed by `evaluation_reporting/` as `MODEL`.