yi_lu/ti_coding_agent_data_prep
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document audit policy and add full kept export

Browse Source
This commit is contained in:
Codex
2026-06-24 22:43:04 +08:00
parent f06e573b04
commit 28b839eff0
4 changed files with 454 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
README.md
View File

@@ -113,13 +113,41 @@ python scripts/filtering/audit_native_traces.py \
python scripts/repurposing/coarse_decompose.py
```
构建 5k ModelScope-SWIFT training probe：
### 模式 A：全量 hard-filter-kept 训练数据
该模式会扫描 Open-SWE-Traces 四个 split 的全部样本，使用
`scripts/filtering/audit_native_traces.py` 中同一套 hard filter 规则，导出所有未触发
hard filter 的样本。按当前数据和规则，预期规模约为 190k 条。
全量导出使用流式 JSONL 写入，避免把约 190k 条长 trajectory 全部放进内存：
```bash
python scripts/repurposing/build_swift_full_kept.py \
--input-root data/Open-SWE-Traces \
--output-dir runs/training_full_kept_swift
```
如果需要同时写 gzip：
```bash
python scripts/repurposing/build_swift_full_kept.py \
--input-root data/Open-SWE-Traces \
--output-dir runs/training_full_kept_swift \
--write-gzip
```
### 模式 B：5k+500 probe/测试数据
该模式只用于快速训练链路和数据质量 probing，不代表全量训练集。
构建 5k ModelScope-SWIFT training probe，每个 source config 取 1,250 条 hard-filter-kept 样本：
```bash
python scripts/repurposing/build_swift_training_probe_5k.py
```
构建 500 条 validation split：
构建 500 条 validation split，每个 source config 额外随机取 125 条，并排除 5k train 中的
`trajectory_id`：
```bash
python scripts/repurposing/build_swift_validation_500.py
@@ -144,4 +172,3 @@ python scripts/repurposing/convert_openswe_to_pi_mono.py \
## 输出产物
`runs/` 和 `data/` 默认不进 git。推荐把所有大文件、parquet、jsonl、token 统计、audit report 都留在这两个目录下。

10
SKILL.md
View File

@@ -90,6 +90,15 @@ Build the balanced 500-row validation probe:
python scripts/repurposing/build_swift_validation_500.py
```
Build the full hard-filter-kept SWIFT training set, approximately 190k rows with the current audit policy:
```bash
python scripts/repurposing/build_swift_full_kept.py --input-root data/Open-SWE-Traces --output-dir runs/training_full_kept_swift
```
Use `build_swift_full_kept.py` for real full-data SFT preparation. Use
`build_swift_training_probe_5k.py` plus `build_swift_validation_500.py` only for small probe runs and pipeline tests.
Try pi-mono-style conversion:
```bash
@@ -107,4 +116,3 @@ For SWIFT exports:
- MiniMax trajectories are treated as thinking-mode data and `reasoning_content` is wrapped with `<think>...</think>`.
- Qwen trajectories are treated as non-thinking-mode data.
- `system`, `user`, and `tool` messages use `loss=false`; assistant messages use `loss=true`.

238
docs/resolved_and_audit_policy.md Normal file
View File

@@ -0,0 +1,238 @@
# Open-SWE-Traces resolved 标签与 audit 评价体系
本文说明 `nvidia/Open-SWE-Traces` 中 `resolved` 字段的含义，以及本仓库在
`scripts/filtering/audit_native_traces.py` 中实现的 trajectory 质量审计规则。
## 全量 resolved 分布
统计时间：2026-06-24
统计数据位置：`data/Open-SWE-Traces/data/*_trajectories/*.parquet`
| split | resolved=1 | resolved=0 | resolved=-1 | total |
| --- | ---: | ---: | ---: | ---: |
| `minimax_m25_openhands_trajectories` | 15,941 | 22,911 | 11,096 | 49,948 |
| `minimax_m25_sweagent_trajectories` | 19,551 | 24,554 | 13,163 | 57,268 |
| `qwen35_openhands_trajectories` | 13,657 | 29,147 | 12,684 | 55,488 |
| `qwen35_sweagent_trajectories` | 16,095 | 18,875 | 9,815 | 44,785 |
| **total** | **65,244** | **95,487** | **46,758** | **207,489** |
整体比例：
| 状态 | 数量 | 占比 |
| --- | ---: | ---: |
| resolved，明确解决 | 65,244 | 31.44% |
| unresolved，明确未解决 | 95,487 | 46.02% |
| unknown/uncertain，不确定或非标准成功状态 | 46,758 | 22.54% |
## resolved 是什么
`resolved` 是 Open-SWE-Traces 数据集自带的上游评测结果标签，不是本仓库的
audit 规则计算出来的字段。
当前数据中可观察到三个取值：
- `resolved == 1`：上游认为该 trajectory 的最终 patch 解决了对应任务。
- `resolved == 0`：上游认为未解决。
- `resolved == -1`：不确定、异常或非标准成功状态。当前脚本不把它解释为成功。
在本仓库的 filtering 逻辑中，只有 `resolved == 1` 被视为明确 solved。代码在
`scripts/filtering/audit_native_traces.py:213-219` 将所有 `resolved != 1` 的样本标记为
`unresolved`：
```python
resolved = int(row.get("resolved") or 0)
if resolved != 1:
issues.append("unresolved")
```
注意：`unresolved` 本身只是 audit flag，不是 hard filter。也就是说，`resolved=0`
或 `resolved=-1` 不会因为这个字段单独被丢弃。原因是失败轨迹中仍可能存在可学习的
定位、诊断、部分修复或验证行为。
## audit flag 是什么
`audit flag` 是本仓库对每条 trajectory 做规则审计后生成的质量标签，主要用于判断
样本是否适合作为 SFT 数据。
入口函数是 `audit_row`，位置是：
- `scripts/filtering/audit_native_traces.py:140-233`
`audit_row` 会检查以下维度：
- tool call JSON 是否能解析：`parse_tool_arguments`，见 `scripts/filtering/audit_native_traces.py:71-84`
- tool 名是否在已知集合中：`KNOWN_TOOLS`，见 `scripts/filtering/audit_native_traces.py:21-31`
- `str_replace_editor` command 是否在已知集合中：`SOURCE_TOOL_COMMANDS`，见 `scripts/filtering/audit_native_traces.py:35-43`
- tool call 和 tool result 是否匹配：见 `scripts/filtering/audit_native_traces.py:144-205`
- trajectory 是否过长：见 `scripts/filtering/audit_native_traces.py:60-63` 和 `208-209`
- 是否存在重复、无推进的工具调用 loop：`detect_unproductive_loop`，见 `scripts/filtering/audit_native_traces.py:125-137` 和 `210-211`
- 未解决样本是否有明显走偏信号：见 `scripts/filtering/audit_native_traces.py:213-219`
- final patch 是否为空、是否是 unified diff、patch 文件是否在 trajectory 中出现过：`patch_empty_or_inconsistent`，见 `scripts/filtering/audit_native_traces.py:104-122`
`audit_row` 返回两类信息：
- `issues`：该样本的全部 audit flags。
- `details`：统计型信息，例如 tool 调用数量、assistant turn 数、总字符数、编辑次数、验证次数和 trajectory 长度。见 `scripts/filtering/audit_native_traces.py:221-233`。
## audit flag 与 hard filter 的区别
不是所有 audit flag 都会导致样本被过滤。真正决定过滤的是
`hard_filter_issues(issues)`，位置是：
- `scripts/filtering/audit_native_traces.py:287-312`
`hard_filter_issues` 会从全部 `issues` 中挑出严重问题。只要 hard issues 非空，
`audit_file` 就会把该样本计入 `filtered_rows`，并把样本写入坏例子采样。对应逻辑见：
- `scripts/filtering/audit_native_traces.py:255-276`
因此：
- `audit_flags` 是全量质量诊断。
- `hard_filter_issues` 是实际过滤依据。
- `resolved != 1` 只会生成 `unresolved` 软标记；单独的 `unresolved` 不会触发 hard filter。
## 当前 hard filter 规则
`hard_filter_issues` 当前包含两类规则：精确匹配和前缀匹配。
前缀匹配，见 `scripts/filtering/audit_native_traces.py:289-292`：
- `unknown_tool_name:*`
- `unknown_str_replace_editor_command:*`
精确匹配，见 `scripts/filtering/audit_native_traces.py:293-308`：
| hard issue | 含义 |
| --- | --- |
| `malformed_tool_call_json` | tool arguments 是字符串但无法解析为 JSON |
| `tool_arguments_not_string_or_object` | tool arguments 既不是字符串也不是对象 |
| `tool_arguments_json_not_object` | tool arguments JSON 能解析，但不是对象 |
| `tool_arguments_polluted_by_markup_or_text` | tool arguments 中混入明显 markup 或自然语言污染 |
| `tool_name_missing_or_invalid` | tool 名缺失或类型不合法 |
| `tool_result_without_pending_tool_call` | 出现 tool result，但没有等待中的 tool call |
| `tool_call_without_tool_result` | 存在未闭合的 tool call |
| `final_patch_empty` | final patch 为空 |
| `resolved_but_patch_empty` | 上游标记 solved，但 final patch 为空 |
| `final_patch_not_unified_diff` | final patch 不像 unified diff |
| `patch_files_not_mentioned_in_trajectory` | patch 涉及文件在 trajectory 文本中完全没出现 |
| `trajectory_too_long` | trajectory turn 数或字符数超过阈值 |
| `repeated_tool_call_loop` | 短窗口内重复相同 tool call，疑似无推进循环 |
| `unresolved_likely_off_track` | 未解决，同时出现过长或重复 loop 等明显走偏信号 |
## 为什么不直接只保留 resolved=1
如果只保留 `resolved=1`，全量数据会从 207,489 条降到 65,244 条，只剩 31.44%。
这对训练会丢掉大量可能有价值的行为轨迹，包括：
- 如何理解任务；
- 如何定位相关文件；
- 如何读取和诊断代码；
- 如何尝试复现问题；
- 如何做部分修复；
- 如何发现验证失败。
因此当前策略更保守：
1. 保留 `resolved` 作为上游结果标签。
2. 用 audit/hard filter 去掉结构错误、工具链错误、patch 不一致、超长 loop、明显走偏等坏样本。
3. 在后续训练或采样阶段，再根据任务目标决定是否偏向 solved 样本，或保留部分 unresolved 轨迹用于过程学习。
## 运行相关统计的命令
## 两种数据导出模式
当前仓库把“全量训练数据筛选”和“小规模 probe/测试数据构建”拆成两个模式，避免把
pipeline smoke test 和正式训练数据混在一起。
### 模式 A：全量 hard-filter-kept 数据
入口：
```bash
python scripts/repurposing/build_swift_full_kept.py \
--input-root data/Open-SWE-Traces \
--output-dir runs/training_full_kept_swift
```
用途：
- 用于正式 SFT 数据准备。
- 扫描四个 source config 的所有 trajectory。
- 对每条样本调用 `audit_row` 和 `hard_filter_issues`。
- 丢弃所有 hard filter 非空的样本。
- 将剩余样本按 ModelScope-SWIFT messages 格式流式写入 `train.jsonl`。
为什么使用流式 JSONL：
- 全量 kept 规模约 190k 条。
- trajectory 很长，不能像 5k probe 那样把所有样本先放入内存再统一写出。
- 默认只写 `train.jsonl` 和 `metadata.json`；如需 gzip，可加 `--write-gzip`。
### 模式 B：5k+500 probe/测试数据
入口：
```bash
python scripts/repurposing/build_swift_training_probe_5k.py
python scripts/repurposing/build_swift_validation_500.py
```
用途：
- 用于训练链路 smoke test、格式验证、上传测试和小规模 probing。
- `build_swift_training_probe_5k.py` 从四个 source config 各取 1,250 条 hard-filter-kept 样本，总计 5,000 条。
- `build_swift_validation_500.py` 从四个 source config 各随机取 125 条，总计 500 条，并排除 train 中已有的 `trajectory_id`。
- 该模式会额外写 parquet/gzip 等方便测试和上传的小规模产物。
关键区别：
| 维度 | 全量 hard-filter-kept 模式 | 5k+500 probe 模式 |
| --- | --- | --- |
| 入口脚本 | `build_swift_full_kept.py` | `build_swift_training_probe_5k.py` + `build_swift_validation_500.py` |
| 目标 | 正式 SFT 数据准备 | 快速测试和 probing |
| 样本规模 | 约 190k kept rows | 5,000 train + 500 validation |
| 采样方式 | 不采样，扫描全量并过滤 | 按四个 source config 均衡采样 |
| 写出方式 | 流式 JSONL，避免爆内存 | 内存聚合后写 JSONL/GZIP/Parquet |
| 是否应代表全量分布 | 是，过滤后保留全量 | 否，只是小规模测试集 |
统计 `resolved` 分布：
```bash
cd /ssd/workspace/yi/ti_coding_agent_data_prep
.venv/bin/python - <<'PY'
from pathlib import Path
from collections import Counter
import json
import pyarrow.parquet as pq
configs = [
"minimax_m25_openhands_trajectories",
"minimax_m25_sweagent_trajectories",
"qwen35_openhands_trajectories",
"qwen35_sweagent_trajectories",
]
root = Path("data/Open-SWE-Traces/data")
out = {}
total = Counter()
for cfg in configs:
c = Counter()
for f in sorted((root / cfg).glob("*.parquet")):
table = pq.read_table(f, columns=["resolved"])
for row in table.to_pylist():
c[str(row.get("resolved"))] += 1
out[cfg] = dict(c)
total.update(c)
out["TOTAL"] = dict(total)
print(json.dumps(out, indent=2, ensure_ascii=False))
PY
```
运行 audit：
```bash
cd /ssd/workspace/yi/ti_coding_agent_data_prep
.venv/bin/python scripts/filtering/audit_native_traces.py \
--input-root data/Open-SWE-Traces \
--output-dir runs/native_trace_audit
```

177
scripts/repurposing/build_swift_full_kept.py Normal file
View File

@@ -0,0 +1,177 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import gzip
import json
import sys
from collections import Counter
from contextlib import ExitStack
from pathlib import Path
from typing import Any, TextIO
import pyarrow.parquet as pq
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT / "scripts" / "filtering"))
import audit_native_traces as audit # noqa: E402
sys.path.insert(0, str(REPO_ROOT / "scripts" / "repurposing"))
import build_swift_training_probe_5k as train_builder # noqa: E402
class AuditArgs:
long_turn_threshold = 300
long_char_threshold = 900_000
repeat_window = 24
repeat_threshold = 10
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Export all hard-filter-kept Open-SWE-Traces rows to ModelScope-SWIFT JSONL."
)
parser.add_argument("--input-root", type=Path, default=Path("data/Open-SWE-Traces"))
parser.add_argument("--output-dir", type=Path, default=Path("runs/training_full_kept_swift"))
parser.add_argument("--batch-size", type=int, default=128)
parser.add_argument("--limit", type=int, default=0, help="Optional total row limit for smoke tests.")
parser.add_argument("--write-gzip", action="store_true", help="Also write train.jsonl.gz.")
return parser.parse_args()
def normalize_json(value: Any) -> Any:
return json.loads(json.dumps(value, ensure_ascii=False, default=str))
def build_item(
row: dict[str, Any],
config: str,
stats: Counter,
issues: list[str],
details: Counter,
) -> dict[str, Any]:
model_family, scaffold, thinking_mode = train_builder.CONFIG_META[config]
messages = train_builder.convert_messages(row.get("trajectory") or [], thinking_mode, stats)
return {
"messages": messages,
"source_dataset": "nvidia/Open-SWE-Traces",
"source_config": config,
"model_family": model_family,
"scaffold": scaffold,
"thinking_mode": thinking_mode,
"instance_id": row.get("instance_id"),
"repo": row.get("repo"),
"language": row.get("language"),
"license": row.get("license"),
"trajectory_id": row.get("trajectory_id"),
"resolved": int(row.get("resolved") or 0),
"model_patch": row.get("model_patch") or "",
"metadata": normalize_json(row.get("metadata") or {}),
"audit_flags": issues,
"audit_details": dict(details),
}
def write_jsonl_row(handle: TextIO, item: dict[str, Any]) -> None:
handle.write(json.dumps(item, ensure_ascii=False, separators=(",", ":")) + "\n")
def main() -> int:
args = parse_args()
args.output_dir.mkdir(parents=True, exist_ok=True)
data_root = args.input_root / "data"
audit_args = AuditArgs()
stats = Counter()
remaining = args.limit or None
with ExitStack() as stack:
jsonl = stack.enter_context((args.output_dir / "train.jsonl").open("w", encoding="utf-8"))
gz = None
if args.write_gzip:
gz = stack.enter_context(gzip.open(args.output_dir / "train.jsonl.gz", "wt", encoding="utf-8"))
for config in audit.CONFIGS:
for file in sorted((data_root / config).glob("*.parquet")):
if remaining is not None and remaining <= 0:
break
parquet = pq.ParquetFile(file)
for batch in parquet.iter_batches(batch_size=args.batch_size):
rows = batch.to_pylist()
if remaining is not None:
rows = rows[:remaining]
for row in rows:
stats[f"seen:{config}"] += 1
issues, _details = audit.audit_row(row, audit_args)
hard = audit.hard_filter_issues(issues)
if hard:
stats[f"filtered:{config}"] += 1
for issue in hard:
stats[f"hard_issue:{issue}"] += 1
continue
item = build_item(row, config, stats, issues, _details)
write_jsonl_row(jsonl, item)
if gz is not None:
write_jsonl_row(gz, item)
stats[f"kept:{config}"] += 1
stats[f"resolved:{config}:{item['resolved']}"] += 1
if remaining is not None:
remaining -= len(rows)
if remaining <= 0:
break
print(
json.dumps(
{
"file": str(file),
"seen": stats[f"seen:{config}"],
"kept": stats[f"kept:{config}"],
"filtered": stats[f"filtered:{config}"],
},
ensure_ascii=False,
),
flush=True,
)
metadata = {
"name": "open-swe-traces-swift-full-kept",
"source_dataset": "nvidia/Open-SWE-Traces",
"format": "modelscope-swift messages JSONL",
"selection": "all rows without hard-filter issues",
"stats": dict(stats),
"thinking_policy": {
"minimax": "reasoning_content is wrapped as <think>...</think> in assistant content",
"qwen": "non-thinking export; reasoning_content is not emitted; unexpected nonempty reasoning is counted",
"tool_response_mask": "tool messages have loss=false",
},
"files": ["train.jsonl", "metadata.json"] + (["train.jsonl.gz"] if args.write_gzip else []),
}
(args.output_dir / "metadata.json").write_text(json.dumps(metadata, indent=2, ensure_ascii=False), encoding="utf-8")
(args.output_dir / "README.md").write_text(build_readme(metadata), encoding="utf-8")
print(json.dumps(metadata, indent=2, ensure_ascii=False))
return 0
def build_readme(metadata: dict[str, Any]) -> str:
return f"""# Open-SWE-Traces Swift Full Kept
This is the full hard-filter-kept export from `nvidia/Open-SWE-Traces` for ModelScope-SWIFT style SFT.
Selection:
- Scan all four Open-SWE-Traces source configs.
- Drop rows with hard-filter issues from `scripts/filtering/audit_native_traces.py`.
- Preserve native scaffold semantics.
- MiniMax rows are exported as thinking examples by wrapping `reasoning_content` in `<think>...</think>`.
- Qwen rows are exported as non-thinking examples.
- Tool responses are included with `loss: false`.
Stats:
```json
{json.dumps(metadata["stats"], indent=2, ensure_ascii=False)}
```
"""
if __name__ == "__main__":
raise SystemExit(main())