yi_lu/laoyao_2b_moe
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Megatron-Bridge pretrain launcher

Browse Source
This commit is contained in:
路易
2026-07-01 23:10:37 +08:00
parent f0f72767f5
commit 2c2b7ccc24
7 changed files with 357 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
model/nemo_megatron/laoyao_2b_moe_nemo_megatron.yaml
View File

@@ -15,8 +15,8 @@ model:
num_attention_heads: 24
num_query_groups: 4
ffn_hidden_size: 4608
max_position_embeddings: 16384
seq_length: 16384
max_position_embeddings: 8192
seq_length: 8192
normalization: rmsnorm
activation: swiglu
position_embedding_type: rope