58 lines
1.6 KiB
Markdown
58 lines
1.6 KiB
Markdown
# Model Building
|
|
|
|
This folder owns construction of the tokenizer-swapped base model.
|
|
|
|
It contains only the tokenizer swap v2 algorithm. Dataset construction, training, and evaluation live in the other workflow folders.
|
|
|
|
## Main Files
|
|
|
|
```text
|
|
build_qwen3_dsv4_remap_checkpoint_v2.py
|
|
run_remap_v2.sh
|
|
```
|
|
|
|
## Inputs
|
|
|
|
The remap script needs:
|
|
|
|
- source Qwen model checkpoint
|
|
- source Qwen tokenizer
|
|
- target DSV4 tokenizer
|
|
|
|
Default paths in `run_remap_v2.sh` are environment-variable driven and can be overridden:
|
|
|
|
```bash
|
|
BASE_MODEL=/path/to/Qwen3-0.6B \
|
|
DSV_TOKENIZER=/path/to/dsv4_tokenizer \
|
|
OUT=/path/to/output_checkpoint \
|
|
bash model_building/run_remap_v2.sh
|
|
```
|
|
|
|
## Output
|
|
|
|
By default, generated checkpoints go to:
|
|
|
|
```text
|
|
model_building/generated_models/
|
|
```
|
|
|
|
This directory is ignored by git. Do not commit checkpoint weights.
|
|
|
|
## Algorithm Summary
|
|
|
|
The v2 remap builds DSV4-sized input embedding and LM-head matrices from the source Qwen checkpoint.
|
|
|
|
For each DSV4 token row, initialization is selected in this priority order:
|
|
|
|
1. Exact same token surface exists in the Qwen vocab.
|
|
2. Functional special-token mapping is available, such as DSV BOS to Qwen `<|im_start|>` and DSV EOS to Qwen EOS.
|
|
3. Byte-level token can be decoded, re-tokenized with Qwen, and initialized by averaging the corresponding Qwen rows.
|
|
4. Raw token decomposition can be tokenized with Qwen and averaged.
|
|
5. Global embedding/head mean fallback.
|
|
|
|
The script writes the remapped checkpoint plus `tokenizer_remap_v2_report.json` for auditability.
|
|
|
|
## Output Contract
|
|
|
|
The output checkpoint is consumed by `model_training/` scripts as `MODEL`.
|