Clean tokenizer swap migration

model_building/README.md

@@ -0,0 +1,57 @@
+# Model Building
+
+This folder owns construction of the tokenizer-swapped base model.
+
+It contains only the tokenizer swap v2 algorithm. Dataset construction, training, and evaluation live in the other workflow folders.
+
+## Main Files
+
+```text
+build_qwen3_dsv4_remap_checkpoint_v2.py
+run_remap_v2.sh
+```
+
+## Inputs
+
+The remap script needs:
+
+- source Qwen model checkpoint
+- source Qwen tokenizer
+- target DSV4 tokenizer
+
+Default paths in `run_remap_v2.sh` are environment-variable driven and can be overridden:
+
+```bash
+BASE_MODEL=/path/to/Qwen3-0.6B \
+DSV_TOKENIZER=/path/to/dsv4_tokenizer \
+OUT=/path/to/output_checkpoint \
+bash model_building/run_remap_v2.sh
+```
+
+## Output
+
+By default, generated checkpoints go to:
+
+```text
+model_building/generated_models/
+```
+
+This directory is ignored by git. Do not commit checkpoint weights.
+
+## Algorithm Summary
+
+The v2 remap builds DSV4-sized input embedding and LM-head matrices from the source Qwen checkpoint.
+
+For each DSV4 token row, initialization is selected in this priority order:
+
+1. Exact same token surface exists in the Qwen vocab.
+2. Functional special-token mapping is available, such as DSV BOS to Qwen `<|im_start|>` and DSV EOS to Qwen EOS.
+3. Byte-level token can be decoded, re-tokenized with Qwen, and initialized by averaging the corresponding Qwen rows.
+4. Raw token decomposition can be tokenized with Qwen and averaged.
+5. Global embedding/head mean fallback.
+
+The script writes the remapped checkpoint plus `tokenizer_remap_v2_report.json` for auditability.
+
+## Output Contract
+
+The output checkpoint is consumed by `model_training/` scripts as `MODEL`.