Document audit policy and add full kept export

README.md

@@ -113,13 +113,41 @@ python scripts/filtering/audit_native_traces.py \
 python scripts/repurposing/coarse_decompose.py
 ```
 
-构建 5k ModelScope-SWIFT training probe：
+### 模式 A：全量 hard-filter-kept 训练数据
+
+该模式会扫描 Open-SWE-Traces 四个 split 的全部样本，使用
+`scripts/filtering/audit_native_traces.py` 中同一套 hard filter 规则，导出所有未触发
+hard filter 的样本。按当前数据和规则，预期规模约为 190k 条。
+
+全量导出使用流式 JSONL 写入，避免把约 190k 条长 trajectory 全部放进内存：
+
+```bash
+python scripts/repurposing/build_swift_full_kept.py \
+  --input-root data/Open-SWE-Traces \
+  --output-dir runs/training_full_kept_swift
+```
+
+如果需要同时写 gzip：
+
+```bash
+python scripts/repurposing/build_swift_full_kept.py \
+  --input-root data/Open-SWE-Traces \
+  --output-dir runs/training_full_kept_swift \
+  --write-gzip
+```
+
+### 模式 B：5k+500 probe/测试数据
+
+该模式只用于快速训练链路和数据质量 probing，不代表全量训练集。
+
+构建 5k ModelScope-SWIFT training probe，每个 source config 取 1,250 条 hard-filter-kept 样本：
 
 ```bash
 python scripts/repurposing/build_swift_training_probe_5k.py
 ```
 
-构建 500 条 validation split：
+构建 500 条 validation split，每个 source config 额外随机取 125 条，并排除 5k train 中的
+`trajectory_id`：
 
 ```bash
 python scripts/repurposing/build_swift_validation_500.py
@@ -144,4 +172,3 @@ python scripts/repurposing/convert_openswe_to_pi_mono.py \
 ## 输出产物
 
 `runs/` 和 `data/` 默认不进 git。推荐把所有大文件、parquet、jsonl、token 统计、audit report 都留在这两个目录下。
-

SKILL.md

@@ -90,6 +90,15 @@ Build the balanced 500-row validation probe:
 python scripts/repurposing/build_swift_validation_500.py
 ```
 
+Build the full hard-filter-kept SWIFT training set, approximately 190k rows with the current audit policy:
+
+```bash
+python scripts/repurposing/build_swift_full_kept.py --input-root data/Open-SWE-Traces --output-dir runs/training_full_kept_swift
+```
+
+Use `build_swift_full_kept.py` for real full-data SFT preparation. Use
+`build_swift_training_probe_5k.py` plus `build_swift_validation_500.py` only for small probe runs and pipeline tests.
+
 Try pi-mono-style conversion:
 
 ```bash
@@ -107,4 +116,3 @@ For SWIFT exports:
 - MiniMax trajectories are treated as thinking-mode data and `reasoning_content` is wrapped with `<think>...</think>`.
 - Qwen trajectories are treated as non-thinking-mode data.
 - `system`, `user`, and `tool` messages use `loss=false`; assistant messages use `loss=true`.
-

docs/resolved_and_audit_policy.md

@@ -0,0 +1,238 @@
+# Open-SWE-Traces resolved 标签与 audit 评价体系
+
+本文说明 `nvidia/Open-SWE-Traces` 中 `resolved` 字段的含义，以及本仓库在
+`scripts/filtering/audit_native_traces.py` 中实现的 trajectory 质量审计规则。
+
+## 全量 resolved 分布
+
+统计时间：2026-06-24  
+统计数据位置：`data/Open-SWE-Traces/data/*_trajectories/*.parquet`
+
+| split | resolved=1 | resolved=0 | resolved=-1 | total |
+| --- | ---: | ---: | ---: | ---: |
+| `minimax_m25_openhands_trajectories` | 15,941 | 22,911 | 11,096 | 49,948 |
+| `minimax_m25_sweagent_trajectories` | 19,551 | 24,554 | 13,163 | 57,268 |
+| `qwen35_openhands_trajectories` | 13,657 | 29,147 | 12,684 | 55,488 |
+| `qwen35_sweagent_trajectories` | 16,095 | 18,875 | 9,815 | 44,785 |
+| **total** | **65,244** | **95,487** | **46,758** | **207,489** |
+
+整体比例：
+
+| 状态 | 数量 | 占比 |
+| --- | ---: | ---: |
+| resolved，明确解决 | 65,244 | 31.44% |
+| unresolved，明确未解决 | 95,487 | 46.02% |
+| unknown/uncertain，不确定或非标准成功状态 | 46,758 | 22.54% |
+
+## resolved 是什么
+
+`resolved` 是 Open-SWE-Traces 数据集自带的上游评测结果标签，不是本仓库的
+audit 规则计算出来的字段。
+
+当前数据中可观察到三个取值：
+
+- `resolved == 1`：上游认为该 trajectory 的最终 patch 解决了对应任务。
+- `resolved == 0`：上游认为未解决。
+- `resolved == -1`：不确定、异常或非标准成功状态。当前脚本不把它解释为成功。
+
+在本仓库的 filtering 逻辑中，只有 `resolved == 1` 被视为明确 solved。代码在
+`scripts/filtering/audit_native_traces.py:213-219` 将所有 `resolved != 1` 的样本标记为
+`unresolved`：
+
+```python
+resolved = int(row.get("resolved") or 0)
+if resolved != 1:
+    issues.append("unresolved")
+```
+
+注意：`unresolved` 本身只是 audit flag，不是 hard filter。也就是说，`resolved=0`
+或 `resolved=-1` 不会因为这个字段单独被丢弃。原因是失败轨迹中仍可能存在可学习的
+定位、诊断、部分修复或验证行为。
+
+## audit flag 是什么
+
+`audit flag` 是本仓库对每条 trajectory 做规则审计后生成的质量标签，主要用于判断
+样本是否适合作为 SFT 数据。
+
+入口函数是 `audit_row`，位置是：
+
+- `scripts/filtering/audit_native_traces.py:140-233`
+
+`audit_row` 会检查以下维度：
+
+- tool call JSON 是否能解析：`parse_tool_arguments`，见 `scripts/filtering/audit_native_traces.py:71-84`
+- tool 名是否在已知集合中：`KNOWN_TOOLS`，见 `scripts/filtering/audit_native_traces.py:21-31`
+- `str_replace_editor` command 是否在已知集合中：`SOURCE_TOOL_COMMANDS`，见 `scripts/filtering/audit_native_traces.py:35-43`
+- tool call 和 tool result 是否匹配：见 `scripts/filtering/audit_native_traces.py:144-205`
+- trajectory 是否过长：见 `scripts/filtering/audit_native_traces.py:60-63` 和 `208-209`
+- 是否存在重复、无推进的工具调用 loop：`detect_unproductive_loop`，见 `scripts/filtering/audit_native_traces.py:125-137` 和 `210-211`
+- 未解决样本是否有明显走偏信号：见 `scripts/filtering/audit_native_traces.py:213-219`
+- final patch 是否为空、是否是 unified diff、patch 文件是否在 trajectory 中出现过：`patch_empty_or_inconsistent`，见 `scripts/filtering/audit_native_traces.py:104-122`
+
+`audit_row` 返回两类信息：
+
+- `issues`：该样本的全部 audit flags。
+- `details`：统计型信息，例如 tool 调用数量、assistant turn 数、总字符数、编辑次数、验证次数和 trajectory 长度。见 `scripts/filtering/audit_native_traces.py:221-233`。
+
+## audit flag 与 hard filter 的区别
+
+不是所有 audit flag 都会导致样本被过滤。真正决定过滤的是
+`hard_filter_issues(issues)`，位置是：
+
+- `scripts/filtering/audit_native_traces.py:287-312`
+
+`hard_filter_issues` 会从全部 `issues` 中挑出严重问题。只要 hard issues 非空，
+`audit_file` 就会把该样本计入 `filtered_rows`，并把样本写入坏例子采样。对应逻辑见：
+
+- `scripts/filtering/audit_native_traces.py:255-276`
+
+因此：
+
+- `audit_flags` 是全量质量诊断。
+- `hard_filter_issues` 是实际过滤依据。
+- `resolved != 1` 只会生成 `unresolved` 软标记；单独的 `unresolved` 不会触发 hard filter。
+
+## 当前 hard filter 规则
+
+`hard_filter_issues` 当前包含两类规则：精确匹配和前缀匹配。
+
+前缀匹配，见 `scripts/filtering/audit_native_traces.py:289-292`：
+
+- `unknown_tool_name:*`
+- `unknown_str_replace_editor_command:*`
+
+精确匹配，见 `scripts/filtering/audit_native_traces.py:293-308`：
+
+| hard issue | 含义 |
+| --- | --- |
+| `malformed_tool_call_json` | tool arguments 是字符串但无法解析为 JSON |
+| `tool_arguments_not_string_or_object` | tool arguments 既不是字符串也不是对象 |
+| `tool_arguments_json_not_object` | tool arguments JSON 能解析，但不是对象 |
+| `tool_arguments_polluted_by_markup_or_text` | tool arguments 中混入明显 markup 或自然语言污染 |
+| `tool_name_missing_or_invalid` | tool 名缺失或类型不合法 |
+| `tool_result_without_pending_tool_call` | 出现 tool result，但没有等待中的 tool call |
+| `tool_call_without_tool_result` | 存在未闭合的 tool call |
+| `final_patch_empty` | final patch 为空 |
+| `resolved_but_patch_empty` | 上游标记 solved，但 final patch 为空 |
+| `final_patch_not_unified_diff` | final patch 不像 unified diff |
+| `patch_files_not_mentioned_in_trajectory` | patch 涉及文件在 trajectory 文本中完全没出现 |
+| `trajectory_too_long` | trajectory turn 数或字符数超过阈值 |
+| `repeated_tool_call_loop` | 短窗口内重复相同 tool call，疑似无推进循环 |
+| `unresolved_likely_off_track` | 未解决，同时出现过长或重复 loop 等明显走偏信号 |
+
+## 为什么不直接只保留 resolved=1
+
+如果只保留 `resolved=1`，全量数据会从 207,489 条降到 65,244 条，只剩 31.44%。
+这对训练会丢掉大量可能有价值的行为轨迹，包括：
+
+- 如何理解任务；
+- 如何定位相关文件；
+- 如何读取和诊断代码；
+- 如何尝试复现问题；
+- 如何做部分修复；
+- 如何发现验证失败。
+
+因此当前策略更保守：
+
+1. 保留 `resolved` 作为上游结果标签。
+2. 用 audit/hard filter 去掉结构错误、工具链错误、patch 不一致、超长 loop、明显走偏等坏样本。
+3. 在后续训练或采样阶段，再根据任务目标决定是否偏向 solved 样本，或保留部分 unresolved 轨迹用于过程学习。
+
+## 运行相关统计的命令
+
+## 两种数据导出模式
+
+当前仓库把“全量训练数据筛选”和“小规模 probe/测试数据构建”拆成两个模式，避免把
+pipeline smoke test 和正式训练数据混在一起。
+
+### 模式 A：全量 hard-filter-kept 数据
+
+入口：
+
+```bash
+python scripts/repurposing/build_swift_full_kept.py \
+  --input-root data/Open-SWE-Traces \
+  --output-dir runs/training_full_kept_swift
+```
+
+用途：
+
+- 用于正式 SFT 数据准备。
+- 扫描四个 source config 的所有 trajectory。
+- 对每条样本调用 `audit_row` 和 `hard_filter_issues`。
+- 丢弃所有 hard filter 非空的样本。
+- 将剩余样本按 ModelScope-SWIFT messages 格式流式写入 `train.jsonl`。
+
+为什么使用流式 JSONL：
+
+- 全量 kept 规模约 190k 条。
+- trajectory 很长，不能像 5k probe 那样把所有样本先放入内存再统一写出。
+- 默认只写 `train.jsonl` 和 `metadata.json`；如需 gzip，可加 `--write-gzip`。
+
+### 模式 B：5k+500 probe/测试数据
+
+入口：
+
+```bash
+python scripts/repurposing/build_swift_training_probe_5k.py
+python scripts/repurposing/build_swift_validation_500.py
+```
+
+用途：
+
+- 用于训练链路 smoke test、格式验证、上传测试和小规模 probing。
+- `build_swift_training_probe_5k.py` 从四个 source config 各取 1,250 条 hard-filter-kept 样本，总计 5,000 条。
+- `build_swift_validation_500.py` 从四个 source config 各随机取 125 条，总计 500 条，并排除 train 中已有的 `trajectory_id`。
+- 该模式会额外写 parquet/gzip 等方便测试和上传的小规模产物。
+
+关键区别：
+
+| 维度 | 全量 hard-filter-kept 模式 | 5k+500 probe 模式 |
+| --- | --- | --- |
+| 入口脚本 | `build_swift_full_kept.py` | `build_swift_training_probe_5k.py` + `build_swift_validation_500.py` |
+| 目标 | 正式 SFT 数据准备 | 快速测试和 probing |
+| 样本规模 | 约 190k kept rows | 5,000 train + 500 validation |
+| 采样方式 | 不采样，扫描全量并过滤 | 按四个 source config 均衡采样 |
+| 写出方式 | 流式 JSONL，避免爆内存 | 内存聚合后写 JSONL/GZIP/Parquet |
+| 是否应代表全量分布 | 是，过滤后保留全量 | 否，只是小规模测试集 |
+
+统计 `resolved` 分布：
+
+```bash
+cd /ssd/workspace/yi/ti_coding_agent_data_prep
+.venv/bin/python - <<'PY'
+from pathlib import Path
+from collections import Counter
+import json
+import pyarrow.parquet as pq
+
+configs = [
+    "minimax_m25_openhands_trajectories",
+    "minimax_m25_sweagent_trajectories",
+    "qwen35_openhands_trajectories",
+    "qwen35_sweagent_trajectories",
+]
+root = Path("data/Open-SWE-Traces/data")
+out = {}
+total = Counter()
+for cfg in configs:
+    c = Counter()
+    for f in sorted((root / cfg).glob("*.parquet")):
+        table = pq.read_table(f, columns=["resolved"])
+        for row in table.to_pylist():
+            c[str(row.get("resolved"))] += 1
+    out[cfg] = dict(c)
+    total.update(c)
+out["TOTAL"] = dict(total)
+print(json.dumps(out, indent=2, ensure_ascii=False))
+PY
+```
+
+运行 audit：
+
+```bash
+cd /ssd/workspace/yi/ti_coding_agent_data_prep
+.venv/bin/python scripts/filtering/audit_native_traces.py \
+  --input-root data/Open-SWE-Traces \
+  --output-dir runs/native_trace_audit
+```

scripts/repurposing/build_swift_full_kept.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import gzip
+import json
+import sys
+from collections import Counter
+from contextlib import ExitStack
+from pathlib import Path
+from typing import Any, TextIO
+
+import pyarrow.parquet as pq
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(REPO_ROOT / "scripts" / "filtering"))
+import audit_native_traces as audit  # noqa: E402
+sys.path.insert(0, str(REPO_ROOT / "scripts" / "repurposing"))
+import build_swift_training_probe_5k as train_builder  # noqa: E402
+
+
+class AuditArgs:
+    long_turn_threshold = 300
+    long_char_threshold = 900_000
+    repeat_window = 24
+    repeat_threshold = 10
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Export all hard-filter-kept Open-SWE-Traces rows to ModelScope-SWIFT JSONL."
+    )
+    parser.add_argument("--input-root", type=Path, default=Path("data/Open-SWE-Traces"))
+    parser.add_argument("--output-dir", type=Path, default=Path("runs/training_full_kept_swift"))
+    parser.add_argument("--batch-size", type=int, default=128)
+    parser.add_argument("--limit", type=int, default=0, help="Optional total row limit for smoke tests.")
+    parser.add_argument("--write-gzip", action="store_true", help="Also write train.jsonl.gz.")
+    return parser.parse_args()
+
+
+def normalize_json(value: Any) -> Any:
+    return json.loads(json.dumps(value, ensure_ascii=False, default=str))
+
+
+def build_item(
+    row: dict[str, Any],
+    config: str,
+    stats: Counter,
+    issues: list[str],
+    details: Counter,
+) -> dict[str, Any]:
+    model_family, scaffold, thinking_mode = train_builder.CONFIG_META[config]
+    messages = train_builder.convert_messages(row.get("trajectory") or [], thinking_mode, stats)
+    return {
+        "messages": messages,
+        "source_dataset": "nvidia/Open-SWE-Traces",
+        "source_config": config,
+        "model_family": model_family,
+        "scaffold": scaffold,
+        "thinking_mode": thinking_mode,
+        "instance_id": row.get("instance_id"),
+        "repo": row.get("repo"),
+        "language": row.get("language"),
+        "license": row.get("license"),
+        "trajectory_id": row.get("trajectory_id"),
+        "resolved": int(row.get("resolved") or 0),
+        "model_patch": row.get("model_patch") or "",
+        "metadata": normalize_json(row.get("metadata") or {}),
+        "audit_flags": issues,
+        "audit_details": dict(details),
+    }
+
+
+def write_jsonl_row(handle: TextIO, item: dict[str, Any]) -> None:
+    handle.write(json.dumps(item, ensure_ascii=False, separators=(",", ":")) + "\n")
+
+
+def main() -> int:
+    args = parse_args()
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+    data_root = args.input_root / "data"
+    audit_args = AuditArgs()
+    stats = Counter()
+    remaining = args.limit or None
+
+    with ExitStack() as stack:
+        jsonl = stack.enter_context((args.output_dir / "train.jsonl").open("w", encoding="utf-8"))
+        gz = None
+        if args.write_gzip:
+            gz = stack.enter_context(gzip.open(args.output_dir / "train.jsonl.gz", "wt", encoding="utf-8"))
+
+        for config in audit.CONFIGS:
+            for file in sorted((data_root / config).glob("*.parquet")):
+                if remaining is not None and remaining <= 0:
+                    break
+                parquet = pq.ParquetFile(file)
+                for batch in parquet.iter_batches(batch_size=args.batch_size):
+                    rows = batch.to_pylist()
+                    if remaining is not None:
+                        rows = rows[:remaining]
+                    for row in rows:
+                        stats[f"seen:{config}"] += 1
+                        issues, _details = audit.audit_row(row, audit_args)
+                        hard = audit.hard_filter_issues(issues)
+                        if hard:
+                            stats[f"filtered:{config}"] += 1
+                            for issue in hard:
+                                stats[f"hard_issue:{issue}"] += 1
+                            continue
+                        item = build_item(row, config, stats, issues, _details)
+                        write_jsonl_row(jsonl, item)
+                        if gz is not None:
+                            write_jsonl_row(gz, item)
+                        stats[f"kept:{config}"] += 1
+                        stats[f"resolved:{config}:{item['resolved']}"] += 1
+
+                    if remaining is not None:
+                        remaining -= len(rows)
+                        if remaining <= 0:
+                            break
+
+                print(
+                    json.dumps(
+                        {
+                            "file": str(file),
+                            "seen": stats[f"seen:{config}"],
+                            "kept": stats[f"kept:{config}"],
+                            "filtered": stats[f"filtered:{config}"],
+                        },
+                        ensure_ascii=False,
+                    ),
+                    flush=True,
+                )
+
+    metadata = {
+        "name": "open-swe-traces-swift-full-kept",
+        "source_dataset": "nvidia/Open-SWE-Traces",
+        "format": "modelscope-swift messages JSONL",
+        "selection": "all rows without hard-filter issues",
+        "stats": dict(stats),
+        "thinking_policy": {
+            "minimax": "reasoning_content is wrapped as <think>...</think> in assistant content",
+            "qwen": "non-thinking export; reasoning_content is not emitted; unexpected nonempty reasoning is counted",
+            "tool_response_mask": "tool messages have loss=false",
+        },
+        "files": ["train.jsonl", "metadata.json"] + (["train.jsonl.gz"] if args.write_gzip else []),
+    }
+    (args.output_dir / "metadata.json").write_text(json.dumps(metadata, indent=2, ensure_ascii=False), encoding="utf-8")
+    (args.output_dir / "README.md").write_text(build_readme(metadata), encoding="utf-8")
+    print(json.dumps(metadata, indent=2, ensure_ascii=False))
+    return 0
+
+
+def build_readme(metadata: dict[str, Any]) -> str:
+    return f"""# Open-SWE-Traces Swift Full Kept
+
+This is the full hard-filter-kept export from `nvidia/Open-SWE-Traces` for ModelScope-SWIFT style SFT.
+
+Selection:
+
+- Scan all four Open-SWE-Traces source configs.
+- Drop rows with hard-filter issues from `scripts/filtering/audit_native_traces.py`.
+- Preserve native scaffold semantics.
+- MiniMax rows are exported as thinking examples by wrapping `reasoning_content` in `<think>...</think>`.
+- Qwen rows are exported as non-thinking examples.
+- Tool responses are included with `loss: false`.
+
+Stats:
+
+```json
+{json.dumps(metadata["stats"], indent=2, ensure_ascii=False)}
+```
+"""
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())