Model Training

This folder owns full-parameter training recipes for the final experiments.

It does not build datasets and does not perform tokenizer remapping. It consumes artifacts produced by dataset_building/ and model_building/.

Training Implementations

train_dsv4_tokenized_full_sft.py
train_cpt_packed_full.py

train_dsv4_tokenized_full_sft.py trains on pre-tokenized chat JSONL. Prompt tokens are masked with -100; assistant tokens are optimized.

train_cpt_packed_full.py trains next-token prediction over packed CPT blocks.

Final Recipe Scripts

run_sft1m_remap_v2_5epoch.sh
run_sft1m_remap_v2_then_v4_noupsample_5epoch_bsz16.sh
run_cpt1b_seed42_train_eval.sh
run_cpt5b_seed42_train_eval.sh
run_cpt5b_then_sft1m_5epoch.sh

Common Environment Variables

The run scripts are configurable through environment variables:

ROOT     repo root, default /ssd/yi/Tokenizer_Swap
NPROC    number of GPUs/processes, default 8
MODEL    input checkpoint
DATA     packed CPT dataset directory
TRAIN    tokenized SFT train file
EVAL     tokenized SFT validation file
OUT      output checkpoint directory

Example:

ROOT=/ssd/yi/Tokenizer_Swap \
MODEL=/ssd/yi/Tokenizer_Swap/model_building/generated_models/Qwen3-0.6B-DSV4-tokenizer-remap-v2 \
DATA=/ssd/yi/Tokenizer_Swap/dataset_building/generated/cpt_packed_5b_seq8192_seed42_stratified \
OUT=/ssd/yi/Tokenizer_Swap/model_training/checkpoints/cpt5b \
bash model_training/run_cpt5b_seed42_train_eval.sh

Output

Generated checkpoints go under:

model_training/checkpoints/

This directory is ignored by git. Do not commit model weights, optimizer states, or partial checkpoints.

Output Contract

Trained checkpoints are consumed by evaluation_reporting/ as MODEL.

1.8 KiB Raw Blame History