[Fix] Fix bugs and refactor codes in lora for better scalability. (#3652)

Co-authored-by: ShenAo1111 <1377693092@qq.com> Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
2025-02-20 11:51:57 -08:00
parent ac05310098
commit e79f7420be
11 changed files with 459 additions and 200 deletions
@@ -26,6 +26,11 @@ class LoRAConfig:
        self.path = path
        self.hf_config = self.get_lora_config()
        self.target_modules = self.hf_config["target_modules"]
+
+        # TODO: Support more modules
+        if any(module in self.target_modules for module in ["embed_tokens", "lm_head"]):
+            raise ValueError("Not supported yet")
+
        self.r = self.hf_config["r"]
        self.lora_alpha = self.hf_config["lora_alpha"]