Add eight GPU SWIFT and Megatron training scripts
This commit is contained in:
18
SKILL.md
18
SKILL.md
@@ -103,6 +103,7 @@ The default experiment uses:
|
||||
- explicit cosine LR scheduler via `--lr_scheduler_type cosine`
|
||||
- `max_length=262144`
|
||||
- conservative per-device train batch size `1`
|
||||
- default `NPROC_PER_NODE=8`, `CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7`, and `DEEPSPEED=zero2`
|
||||
- checkpoint save every 1000 steps
|
||||
- validation every 1000 steps
|
||||
- TensorBoard logging under `runs/`
|
||||
@@ -116,6 +117,7 @@ export LORA_PER_DEVICE_BATCH_SIZE=2
|
||||
export FULL_PER_DEVICE_BATCH_SIZE=1
|
||||
export QWEN35_9B_LORA_R32_PER_DEVICE_BATCH_SIZE=2
|
||||
export QWEN36_27B_FULL_BF16_PER_DEVICE_BATCH_SIZE=1
|
||||
export MAX_STEPS=10
|
||||
```
|
||||
|
||||
Run-specific variables have the highest precedence, then global `PER_DEVICE_BATCH_SIZE` / `GRAD_ACCUM_STEPS`, then train-type defaults, then the safe default of 1.
|
||||
@@ -128,3 +130,19 @@ export QWEN36_27B_MODEL_ID=<hf-id>
|
||||
export QWEN35_9B_MODEL_PATH=<local-path>
|
||||
export QWEN36_27B_MODEL_PATH=<local-path>
|
||||
```
|
||||
|
||||
## Megatron-SWIFT
|
||||
|
||||
Use `scripts/train_qwen36_27b_megatron_full.sh` for Megatron-SWIFT/MCore-Bridge full SFT. It follows the official `megatron sft` quick-start style and defaults to:
|
||||
|
||||
- `NPROC_PER_NODE=8`
|
||||
- `CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7`
|
||||
- `MEGATRON_MODEL=Qwen/Qwen3.6-27B`
|
||||
- `TENSOR_MODEL_PARALLEL_SIZE=4`
|
||||
- `MICRO_BATCH_SIZE=1`
|
||||
- `GLOBAL_BATCH_SIZE=8`
|
||||
- `MAX_LENGTH=262144`
|
||||
- `LR=1e-5`
|
||||
- `LR_WARMUP_FRACTION=0.1`
|
||||
|
||||
For multi-node or shared-disk runs, keep `MODELSCOPE_CACHE` on shared storage. The default is `/mnt/beegfs/workspace/ti_coding_agent_probe/modelscope_cache`.
|
||||
|
||||
Reference in New Issue
Block a user