Add eight GPU SWIFT and Megatron training scripts

This commit is contained in:
Codex
2026-06-25 20:34:57 +08:00
parent 4f9e0a30a9
commit d2cafeb8a0
5 changed files with 200 additions and 4 deletions

View File

@@ -103,6 +103,7 @@ The default experiment uses:
- explicit cosine LR scheduler via `--lr_scheduler_type cosine`
- `max_length=262144`
- conservative per-device train batch size `1`
- default `NPROC_PER_NODE=8`, `CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7`, and `DEEPSPEED=zero2`
- checkpoint save every 1000 steps
- validation every 1000 steps
- TensorBoard logging under `runs/`
@@ -116,6 +117,7 @@ export LORA_PER_DEVICE_BATCH_SIZE=2
export FULL_PER_DEVICE_BATCH_SIZE=1
export QWEN35_9B_LORA_R32_PER_DEVICE_BATCH_SIZE=2
export QWEN36_27B_FULL_BF16_PER_DEVICE_BATCH_SIZE=1
export MAX_STEPS=10
```
Run-specific variables have the highest precedence, then global `PER_DEVICE_BATCH_SIZE` / `GRAD_ACCUM_STEPS`, then train-type defaults, then the safe default of 1.
@@ -128,3 +130,19 @@ export QWEN36_27B_MODEL_ID=<hf-id>
export QWEN35_9B_MODEL_PATH=<local-path>
export QWEN36_27B_MODEL_PATH=<local-path>
```
## Megatron-SWIFT
Use `scripts/train_qwen36_27b_megatron_full.sh` for Megatron-SWIFT/MCore-Bridge full SFT. It follows the official `megatron sft` quick-start style and defaults to:
- `NPROC_PER_NODE=8`
- `CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7`
- `MEGATRON_MODEL=Qwen/Qwen3.6-27B`
- `TENSOR_MODEL_PARALLEL_SIZE=4`
- `MICRO_BATCH_SIZE=1`
- `GLOBAL_BATCH_SIZE=8`
- `MAX_LENGTH=262144`
- `LR=1e-5`
- `LR_WARMUP_FRACTION=0.1`
For multi-node or shared-disk runs, keep `MODELSCOPE_CACHE` on shared storage. The default is `/mnt/beegfs/workspace/ti_coding_agent_probe/modelscope_cache`.