Initialize rebalanced 2B MoE project

training/README.md

@@ -0,0 +1,15 @@
+# Training
+
+训练迁移目标：从旧的 `laoyaomodel.training` 手写训练循环迁移到 NeMo/Megatron。
+
+当前 recipe：
+
+- 200B token 预训练
+- bf16
+- seq length 16384
+- AdamW
+- warmup ratio 0.01
+- 每 1000 step 保存和验证
+- validation 指标：nll、perplexity、accuracy
+
+accuracy 需要基于 `dataset/val/data/heldout_2p8k.jsonl` 写任务 adapter；nll/perplexity 可以直接用 `heldout_2p8k_sft_prompt_completion.jsonl` 或 `heldout_2p8k_packed_text.jsonl`。

training/nemo_megatron/README.md

@@ -0,0 +1,5 @@
+# NeMo/Megatron Training Recipe
+
+`pretrain_2b_moe_200b.yaml` 保存训练 recipe。它记录数据路径、模型配置路径、优化器、batch、保存频率和验证频率。
+
+`../../scripts/train_nemo_megatron_2b_moe.sh` 是拉起训练的 shell 入口。当前脚本先做路径检查并预留 NeMo Docker 启动点，下一步需要按 g0033 上最终确认的 NeMo/Megatron image 和 launcher 命令接线。

training/nemo_megatron/pretrain_2b_moe_200b.yaml

@@ -0,0 +1,33 @@
+experiment:
+  name: laoyao_2b_moe_rebalanced_200b
+  seed: 42
+  output_dir: /mnt/beegfs/yi/laoyao_2b_moe/runs/laoyao_2b_moe_rebalanced_200b
+
+paths:
+  model_config: /mnt/beegfs/yi/laoyao_2b_moe/model/nemo_megatron/laoyao_2b_moe_nemo_megatron.yaml
+  train_data: /mnt/beegfs/yi/laoyao_2b_moe/dataset/pretrain/data/pretrain_rebalanced_web40_edu20_chinese10_science10_logic10_math5_code5_200b_v1_20260701
+  validation_messages: /mnt/beegfs/yi/laoyao_2b_moe/dataset/val/data/heldout_2p8k.jsonl
+  validation_sft: /mnt/beegfs/yi/laoyao_2b_moe/dataset/val/data/heldout_2p8k_sft_prompt_completion.jsonl
+  validation_packed_text: /mnt/beegfs/yi/laoyao_2b_moe/dataset/val/data/heldout_2p8k_packed_text.jsonl
+
+training:
+  target_tokens: 200000000000
+  global_batch_size: 1024
+  micro_batch_size: 1
+  seq_length: 16384
+  optimizer: adamw
+  learning_rate: 3.0e-4
+  min_learning_rate: 5.0e-6
+  weight_decay: 0.1
+  warmup_ratio: 0.01
+  lr_schedule: cosine
+  grad_clip: 1.0
+  log_every: 10
+  eval_every: 1000
+  save_every: 1000
+  eval_metrics: [nll, perplexity, accuracy]
+
+notes:
+  - Validation accuracy requires a task adapter over the 2.8K messages file.
+  - Validation nll/perplexity can consume prompt/completion or packed_text forms.
+  - Keep generated checkpoints and the 200B corpus out of git.