diff --git a/README.md b/README.md index 6b04dcd..0372bb8 100644 --- a/README.md +++ b/README.md @@ -113,13 +113,41 @@ python scripts/filtering/audit_native_traces.py \ python scripts/repurposing/coarse_decompose.py ``` -构建 5k ModelScope-SWIFT training probe: +### 模式 A:全量 hard-filter-kept 训练数据 + +该模式会扫描 Open-SWE-Traces 四个 split 的全部样本,使用 +`scripts/filtering/audit_native_traces.py` 中同一套 hard filter 规则,导出所有未触发 +hard filter 的样本。按当前数据和规则,预期规模约为 190k 条。 + +全量导出使用流式 JSONL 写入,避免把约 190k 条长 trajectory 全部放进内存: + +```bash +python scripts/repurposing/build_swift_full_kept.py \ + --input-root data/Open-SWE-Traces \ + --output-dir runs/training_full_kept_swift +``` + +如果需要同时写 gzip: + +```bash +python scripts/repurposing/build_swift_full_kept.py \ + --input-root data/Open-SWE-Traces \ + --output-dir runs/training_full_kept_swift \ + --write-gzip +``` + +### 模式 B:5k+500 probe/测试数据 + +该模式只用于快速训练链路和数据质量 probing,不代表全量训练集。 + +构建 5k ModelScope-SWIFT training probe,每个 source config 取 1,250 条 hard-filter-kept 样本: ```bash python scripts/repurposing/build_swift_training_probe_5k.py ``` -构建 500 条 validation split: +构建 500 条 validation split,每个 source config 额外随机取 125 条,并排除 5k train 中的 +`trajectory_id`: ```bash python scripts/repurposing/build_swift_validation_500.py @@ -144,4 +172,3 @@ python scripts/repurposing/convert_openswe_to_pi_mono.py \ ## 输出产物 `runs/` 和 `data/` 默认不进 git。推荐把所有大文件、parquet、jsonl、token 统计、audit report 都留在这两个目录下。 - diff --git a/SKILL.md b/SKILL.md index 825eccc..c713d90 100644 --- a/SKILL.md +++ b/SKILL.md @@ -90,6 +90,15 @@ Build the balanced 500-row validation probe: python scripts/repurposing/build_swift_validation_500.py ``` +Build the full hard-filter-kept SWIFT training set, approximately 190k rows with the current audit policy: + +```bash +python scripts/repurposing/build_swift_full_kept.py --input-root data/Open-SWE-Traces --output-dir runs/training_full_kept_swift +``` + +Use `build_swift_full_kept.py` for real full-data SFT preparation. Use +`build_swift_training_probe_5k.py` plus `build_swift_validation_500.py` only for small probe runs and pipeline tests. + Try pi-mono-style conversion: ```bash @@ -107,4 +116,3 @@ For SWIFT exports: - MiniMax trajectories are treated as thinking-mode data and `reasoning_content` is wrapped with `...`. - Qwen trajectories are treated as non-thinking-mode data. - `system`, `user`, and `tool` messages use `loss=false`; assistant messages use `loss=true`. - diff --git a/docs/resolved_and_audit_policy.md b/docs/resolved_and_audit_policy.md new file mode 100644 index 0000000..d367692 --- /dev/null +++ b/docs/resolved_and_audit_policy.md @@ -0,0 +1,238 @@ +# Open-SWE-Traces resolved 标签与 audit 评价体系 + +本文说明 `nvidia/Open-SWE-Traces` 中 `resolved` 字段的含义,以及本仓库在 +`scripts/filtering/audit_native_traces.py` 中实现的 trajectory 质量审计规则。 + +## 全量 resolved 分布 + +统计时间:2026-06-24 +统计数据位置:`data/Open-SWE-Traces/data/*_trajectories/*.parquet` + +| split | resolved=1 | resolved=0 | resolved=-1 | total | +| --- | ---: | ---: | ---: | ---: | +| `minimax_m25_openhands_trajectories` | 15,941 | 22,911 | 11,096 | 49,948 | +| `minimax_m25_sweagent_trajectories` | 19,551 | 24,554 | 13,163 | 57,268 | +| `qwen35_openhands_trajectories` | 13,657 | 29,147 | 12,684 | 55,488 | +| `qwen35_sweagent_trajectories` | 16,095 | 18,875 | 9,815 | 44,785 | +| **total** | **65,244** | **95,487** | **46,758** | **207,489** | + +整体比例: + +| 状态 | 数量 | 占比 | +| --- | ---: | ---: | +| resolved,明确解决 | 65,244 | 31.44% | +| unresolved,明确未解决 | 95,487 | 46.02% | +| unknown/uncertain,不确定或非标准成功状态 | 46,758 | 22.54% | + +## resolved 是什么 + +`resolved` 是 Open-SWE-Traces 数据集自带的上游评测结果标签,不是本仓库的 +audit 规则计算出来的字段。 + +当前数据中可观察到三个取值: + +- `resolved == 1`:上游认为该 trajectory 的最终 patch 解决了对应任务。 +- `resolved == 0`:上游认为未解决。 +- `resolved == -1`:不确定、异常或非标准成功状态。当前脚本不把它解释为成功。 + +在本仓库的 filtering 逻辑中,只有 `resolved == 1` 被视为明确 solved。代码在 +`scripts/filtering/audit_native_traces.py:213-219` 将所有 `resolved != 1` 的样本标记为 +`unresolved`: + +```python +resolved = int(row.get("resolved") or 0) +if resolved != 1: + issues.append("unresolved") +``` + +注意:`unresolved` 本身只是 audit flag,不是 hard filter。也就是说,`resolved=0` +或 `resolved=-1` 不会因为这个字段单独被丢弃。原因是失败轨迹中仍可能存在可学习的 +定位、诊断、部分修复或验证行为。 + +## audit flag 是什么 + +`audit flag` 是本仓库对每条 trajectory 做规则审计后生成的质量标签,主要用于判断 +样本是否适合作为 SFT 数据。 + +入口函数是 `audit_row`,位置是: + +- `scripts/filtering/audit_native_traces.py:140-233` + +`audit_row` 会检查以下维度: + +- tool call JSON 是否能解析:`parse_tool_arguments`,见 `scripts/filtering/audit_native_traces.py:71-84` +- tool 名是否在已知集合中:`KNOWN_TOOLS`,见 `scripts/filtering/audit_native_traces.py:21-31` +- `str_replace_editor` command 是否在已知集合中:`SOURCE_TOOL_COMMANDS`,见 `scripts/filtering/audit_native_traces.py:35-43` +- tool call 和 tool result 是否匹配:见 `scripts/filtering/audit_native_traces.py:144-205` +- trajectory 是否过长:见 `scripts/filtering/audit_native_traces.py:60-63` 和 `208-209` +- 是否存在重复、无推进的工具调用 loop:`detect_unproductive_loop`,见 `scripts/filtering/audit_native_traces.py:125-137` 和 `210-211` +- 未解决样本是否有明显走偏信号:见 `scripts/filtering/audit_native_traces.py:213-219` +- final patch 是否为空、是否是 unified diff、patch 文件是否在 trajectory 中出现过:`patch_empty_or_inconsistent`,见 `scripts/filtering/audit_native_traces.py:104-122` + +`audit_row` 返回两类信息: + +- `issues`:该样本的全部 audit flags。 +- `details`:统计型信息,例如 tool 调用数量、assistant turn 数、总字符数、编辑次数、验证次数和 trajectory 长度。见 `scripts/filtering/audit_native_traces.py:221-233`。 + +## audit flag 与 hard filter 的区别 + +不是所有 audit flag 都会导致样本被过滤。真正决定过滤的是 +`hard_filter_issues(issues)`,位置是: + +- `scripts/filtering/audit_native_traces.py:287-312` + +`hard_filter_issues` 会从全部 `issues` 中挑出严重问题。只要 hard issues 非空, +`audit_file` 就会把该样本计入 `filtered_rows`,并把样本写入坏例子采样。对应逻辑见: + +- `scripts/filtering/audit_native_traces.py:255-276` + +因此: + +- `audit_flags` 是全量质量诊断。 +- `hard_filter_issues` 是实际过滤依据。 +- `resolved != 1` 只会生成 `unresolved` 软标记;单独的 `unresolved` 不会触发 hard filter。 + +## 当前 hard filter 规则 + +`hard_filter_issues` 当前包含两类规则:精确匹配和前缀匹配。 + +前缀匹配,见 `scripts/filtering/audit_native_traces.py:289-292`: + +- `unknown_tool_name:*` +- `unknown_str_replace_editor_command:*` + +精确匹配,见 `scripts/filtering/audit_native_traces.py:293-308`: + +| hard issue | 含义 | +| --- | --- | +| `malformed_tool_call_json` | tool arguments 是字符串但无法解析为 JSON | +| `tool_arguments_not_string_or_object` | tool arguments 既不是字符串也不是对象 | +| `tool_arguments_json_not_object` | tool arguments JSON 能解析,但不是对象 | +| `tool_arguments_polluted_by_markup_or_text` | tool arguments 中混入明显 markup 或自然语言污染 | +| `tool_name_missing_or_invalid` | tool 名缺失或类型不合法 | +| `tool_result_without_pending_tool_call` | 出现 tool result,但没有等待中的 tool call | +| `tool_call_without_tool_result` | 存在未闭合的 tool call | +| `final_patch_empty` | final patch 为空 | +| `resolved_but_patch_empty` | 上游标记 solved,但 final patch 为空 | +| `final_patch_not_unified_diff` | final patch 不像 unified diff | +| `patch_files_not_mentioned_in_trajectory` | patch 涉及文件在 trajectory 文本中完全没出现 | +| `trajectory_too_long` | trajectory turn 数或字符数超过阈值 | +| `repeated_tool_call_loop` | 短窗口内重复相同 tool call,疑似无推进循环 | +| `unresolved_likely_off_track` | 未解决,同时出现过长或重复 loop 等明显走偏信号 | + +## 为什么不直接只保留 resolved=1 + +如果只保留 `resolved=1`,全量数据会从 207,489 条降到 65,244 条,只剩 31.44%。 +这对训练会丢掉大量可能有价值的行为轨迹,包括: + +- 如何理解任务; +- 如何定位相关文件; +- 如何读取和诊断代码; +- 如何尝试复现问题; +- 如何做部分修复; +- 如何发现验证失败。 + +因此当前策略更保守: + +1. 保留 `resolved` 作为上游结果标签。 +2. 用 audit/hard filter 去掉结构错误、工具链错误、patch 不一致、超长 loop、明显走偏等坏样本。 +3. 在后续训练或采样阶段,再根据任务目标决定是否偏向 solved 样本,或保留部分 unresolved 轨迹用于过程学习。 + +## 运行相关统计的命令 + +## 两种数据导出模式 + +当前仓库把“全量训练数据筛选”和“小规模 probe/测试数据构建”拆成两个模式,避免把 +pipeline smoke test 和正式训练数据混在一起。 + +### 模式 A:全量 hard-filter-kept 数据 + +入口: + +```bash +python scripts/repurposing/build_swift_full_kept.py \ + --input-root data/Open-SWE-Traces \ + --output-dir runs/training_full_kept_swift +``` + +用途: + +- 用于正式 SFT 数据准备。 +- 扫描四个 source config 的所有 trajectory。 +- 对每条样本调用 `audit_row` 和 `hard_filter_issues`。 +- 丢弃所有 hard filter 非空的样本。 +- 将剩余样本按 ModelScope-SWIFT messages 格式流式写入 `train.jsonl`。 + +为什么使用流式 JSONL: + +- 全量 kept 规模约 190k 条。 +- trajectory 很长,不能像 5k probe 那样把所有样本先放入内存再统一写出。 +- 默认只写 `train.jsonl` 和 `metadata.json`;如需 gzip,可加 `--write-gzip`。 + +### 模式 B:5k+500 probe/测试数据 + +入口: + +```bash +python scripts/repurposing/build_swift_training_probe_5k.py +python scripts/repurposing/build_swift_validation_500.py +``` + +用途: + +- 用于训练链路 smoke test、格式验证、上传测试和小规模 probing。 +- `build_swift_training_probe_5k.py` 从四个 source config 各取 1,250 条 hard-filter-kept 样本,总计 5,000 条。 +- `build_swift_validation_500.py` 从四个 source config 各随机取 125 条,总计 500 条,并排除 train 中已有的 `trajectory_id`。 +- 该模式会额外写 parquet/gzip 等方便测试和上传的小规模产物。 + +关键区别: + +| 维度 | 全量 hard-filter-kept 模式 | 5k+500 probe 模式 | +| --- | --- | --- | +| 入口脚本 | `build_swift_full_kept.py` | `build_swift_training_probe_5k.py` + `build_swift_validation_500.py` | +| 目标 | 正式 SFT 数据准备 | 快速测试和 probing | +| 样本规模 | 约 190k kept rows | 5,000 train + 500 validation | +| 采样方式 | 不采样,扫描全量并过滤 | 按四个 source config 均衡采样 | +| 写出方式 | 流式 JSONL,避免爆内存 | 内存聚合后写 JSONL/GZIP/Parquet | +| 是否应代表全量分布 | 是,过滤后保留全量 | 否,只是小规模测试集 | + +统计 `resolved` 分布: + +```bash +cd /ssd/workspace/yi/ti_coding_agent_data_prep +.venv/bin/python - <<'PY' +from pathlib import Path +from collections import Counter +import json +import pyarrow.parquet as pq + +configs = [ + "minimax_m25_openhands_trajectories", + "minimax_m25_sweagent_trajectories", + "qwen35_openhands_trajectories", + "qwen35_sweagent_trajectories", +] +root = Path("data/Open-SWE-Traces/data") +out = {} +total = Counter() +for cfg in configs: + c = Counter() + for f in sorted((root / cfg).glob("*.parquet")): + table = pq.read_table(f, columns=["resolved"]) + for row in table.to_pylist(): + c[str(row.get("resolved"))] += 1 + out[cfg] = dict(c) + total.update(c) +out["TOTAL"] = dict(total) +print(json.dumps(out, indent=2, ensure_ascii=False)) +PY +``` + +运行 audit: + +```bash +cd /ssd/workspace/yi/ti_coding_agent_data_prep +.venv/bin/python scripts/filtering/audit_native_traces.py \ + --input-root data/Open-SWE-Traces \ + --output-dir runs/native_trace_audit +``` diff --git a/scripts/repurposing/build_swift_full_kept.py b/scripts/repurposing/build_swift_full_kept.py new file mode 100644 index 0000000..7c08d06 --- /dev/null +++ b/scripts/repurposing/build_swift_full_kept.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import gzip +import json +import sys +from collections import Counter +from contextlib import ExitStack +from pathlib import Path +from typing import Any, TextIO + +import pyarrow.parquet as pq + +REPO_ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO_ROOT / "scripts" / "filtering")) +import audit_native_traces as audit # noqa: E402 +sys.path.insert(0, str(REPO_ROOT / "scripts" / "repurposing")) +import build_swift_training_probe_5k as train_builder # noqa: E402 + + +class AuditArgs: + long_turn_threshold = 300 + long_char_threshold = 900_000 + repeat_window = 24 + repeat_threshold = 10 + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Export all hard-filter-kept Open-SWE-Traces rows to ModelScope-SWIFT JSONL." + ) + parser.add_argument("--input-root", type=Path, default=Path("data/Open-SWE-Traces")) + parser.add_argument("--output-dir", type=Path, default=Path("runs/training_full_kept_swift")) + parser.add_argument("--batch-size", type=int, default=128) + parser.add_argument("--limit", type=int, default=0, help="Optional total row limit for smoke tests.") + parser.add_argument("--write-gzip", action="store_true", help="Also write train.jsonl.gz.") + return parser.parse_args() + + +def normalize_json(value: Any) -> Any: + return json.loads(json.dumps(value, ensure_ascii=False, default=str)) + + +def build_item( + row: dict[str, Any], + config: str, + stats: Counter, + issues: list[str], + details: Counter, +) -> dict[str, Any]: + model_family, scaffold, thinking_mode = train_builder.CONFIG_META[config] + messages = train_builder.convert_messages(row.get("trajectory") or [], thinking_mode, stats) + return { + "messages": messages, + "source_dataset": "nvidia/Open-SWE-Traces", + "source_config": config, + "model_family": model_family, + "scaffold": scaffold, + "thinking_mode": thinking_mode, + "instance_id": row.get("instance_id"), + "repo": row.get("repo"), + "language": row.get("language"), + "license": row.get("license"), + "trajectory_id": row.get("trajectory_id"), + "resolved": int(row.get("resolved") or 0), + "model_patch": row.get("model_patch") or "", + "metadata": normalize_json(row.get("metadata") or {}), + "audit_flags": issues, + "audit_details": dict(details), + } + + +def write_jsonl_row(handle: TextIO, item: dict[str, Any]) -> None: + handle.write(json.dumps(item, ensure_ascii=False, separators=(",", ":")) + "\n") + + +def main() -> int: + args = parse_args() + args.output_dir.mkdir(parents=True, exist_ok=True) + data_root = args.input_root / "data" + audit_args = AuditArgs() + stats = Counter() + remaining = args.limit or None + + with ExitStack() as stack: + jsonl = stack.enter_context((args.output_dir / "train.jsonl").open("w", encoding="utf-8")) + gz = None + if args.write_gzip: + gz = stack.enter_context(gzip.open(args.output_dir / "train.jsonl.gz", "wt", encoding="utf-8")) + + for config in audit.CONFIGS: + for file in sorted((data_root / config).glob("*.parquet")): + if remaining is not None and remaining <= 0: + break + parquet = pq.ParquetFile(file) + for batch in parquet.iter_batches(batch_size=args.batch_size): + rows = batch.to_pylist() + if remaining is not None: + rows = rows[:remaining] + for row in rows: + stats[f"seen:{config}"] += 1 + issues, _details = audit.audit_row(row, audit_args) + hard = audit.hard_filter_issues(issues) + if hard: + stats[f"filtered:{config}"] += 1 + for issue in hard: + stats[f"hard_issue:{issue}"] += 1 + continue + item = build_item(row, config, stats, issues, _details) + write_jsonl_row(jsonl, item) + if gz is not None: + write_jsonl_row(gz, item) + stats[f"kept:{config}"] += 1 + stats[f"resolved:{config}:{item['resolved']}"] += 1 + + if remaining is not None: + remaining -= len(rows) + if remaining <= 0: + break + + print( + json.dumps( + { + "file": str(file), + "seen": stats[f"seen:{config}"], + "kept": stats[f"kept:{config}"], + "filtered": stats[f"filtered:{config}"], + }, + ensure_ascii=False, + ), + flush=True, + ) + + metadata = { + "name": "open-swe-traces-swift-full-kept", + "source_dataset": "nvidia/Open-SWE-Traces", + "format": "modelscope-swift messages JSONL", + "selection": "all rows without hard-filter issues", + "stats": dict(stats), + "thinking_policy": { + "minimax": "reasoning_content is wrapped as ... in assistant content", + "qwen": "non-thinking export; reasoning_content is not emitted; unexpected nonempty reasoning is counted", + "tool_response_mask": "tool messages have loss=false", + }, + "files": ["train.jsonl", "metadata.json"] + (["train.jsonl.gz"] if args.write_gzip else []), + } + (args.output_dir / "metadata.json").write_text(json.dumps(metadata, indent=2, ensure_ascii=False), encoding="utf-8") + (args.output_dir / "README.md").write_text(build_readme(metadata), encoding="utf-8") + print(json.dumps(metadata, indent=2, ensure_ascii=False)) + return 0 + + +def build_readme(metadata: dict[str, Any]) -> str: + return f"""# Open-SWE-Traces Swift Full Kept + +This is the full hard-filter-kept export from `nvidia/Open-SWE-Traces` for ModelScope-SWIFT style SFT. + +Selection: + +- Scan all four Open-SWE-Traces source configs. +- Drop rows with hard-filter issues from `scripts/filtering/audit_native_traces.py`. +- Preserve native scaffold semantics. +- MiniMax rows are exported as thinking examples by wrapping `reasoning_content` in `...`. +- Qwen rows are exported as non-thinking examples. +- Tool responses are included with `loss: false`. + +Stats: + +```json +{json.dumps(metadata["stats"], indent=2, ensure_ascii=False)} +``` +""" + + +if __name__ == "__main__": + raise SystemExit(main())