1.6 KiB
Model Building
This folder owns construction of the tokenizer-swapped base model.
It contains only the tokenizer swap v2 algorithm. Dataset construction, training, and evaluation live in the other workflow folders.
Main Files
build_qwen3_dsv4_remap_checkpoint_v2.py
run_remap_v2.sh
Inputs
The remap script needs:
- source Qwen model checkpoint
- source Qwen tokenizer
- target DSV4 tokenizer
Default paths in run_remap_v2.sh are environment-variable driven and can be overridden:
BASE_MODEL=/path/to/Qwen3-0.6B \
DSV_TOKENIZER=/path/to/dsv4_tokenizer \
OUT=/path/to/output_checkpoint \
bash model_building/run_remap_v2.sh
Output
By default, generated checkpoints go to:
model_building/generated_models/
This directory is ignored by git. Do not commit checkpoint weights.
Algorithm Summary
The v2 remap builds DSV4-sized input embedding and LM-head matrices from the source Qwen checkpoint.
For each DSV4 token row, initialization is selected in this priority order:
- Exact same token surface exists in the Qwen vocab.
- Functional special-token mapping is available, such as DSV BOS to Qwen
<|im_start|>and DSV EOS to Qwen EOS. - Byte-level token can be decoded, re-tokenized with Qwen, and initialized by averaging the corresponding Qwen rows.
- Raw token decomposition can be tokenized with Qwen and averaged.
- Global embedding/head mean fallback.
The script writes the remapped checkpoint plus tokenizer_remap_v2_report.json for auditability.
Output Contract
The output checkpoint is consumed by model_training/ scripts as MODEL.