Organize Open-SWE-Traces data prep project

2026-06-24 22:34:26 +08:00
commit f06e573b04
27 changed files with 3264 additions and 0 deletions
@@ -0,0 +1,110 @@
+# TI Coding Agent Data Prep Skill
+
+Use this repo for Open-SWE-Traces data preparation before coding-agent SFT experiments.
+
+## Purpose
+
+This project organizes the existing Open-SWE-Traces probing, native-trace filtering, and trajectory repurposing scripts into a Python project. It is intended for work on `nvidia/Open-SWE-Traces`, especially:
+
+- inspecting dataset structure and source splits,
+- auditing trajectories for SFT-quality hard filters,
+- measuring Qwen tokenizer/token statistics,
+- decomposing trajectories into task-oriented phases,
+- exporting balanced ModelScope-SWIFT training/validation probes,
+- experimenting with pi-mono-style conversion.
+
+## Repository Layout
+
+- `scripts/probing/`: dataset inspection, unique-count reports, tokenizer comparison, and token counting.
+- `scripts/filtering/`: native trajectory audit and hard filtering.
+- `scripts/repurposing/`: subproblem decomposition, coarse task-stage conversion, SWIFT export, and pi-mono conversion.
+- `src/ti_coding_agent_data_prep/openswe/`: shared constants and path helpers.
+- `data/`: local datasets; ignored by git.
+- `runs/`: script outputs; ignored by git.
+- `docs/`: retained legacy notes from the original probe workspace.
+
+## Expected Data Location
+
+By default scripts read:
+
+```text
+data/Open-SWE-Traces
+```
+
+On B300, prefer a symlink instead of copying the dataset:
+
+```bash
+ln -s /ssd/workspace/yi/openswetraces_probe/Open-SWE-Traces data/Open-SWE-Traces
+```
+
+## Environment
+
+Use a repo-local environment. On B300, set proxy variables before installing or downloading:
+
+```bash
+export http_proxy=http://100.72.0.101:8888
+export https_proxy=http://100.72.0.101:8888
+export HTTP_PROXY=http://100.72.0.101:8888
+export HTTPS_PROXY=http://100.72.0.101:8888
+export HF_ENDPOINT=https://hf-mirror.com
+uv venv .venv --python 3.10
+source .venv/bin/activate
+uv pip install -e '.[dev]'
+```
+
+## Main Entrypoints
+
+Probe one sample per split:
+
+```bash
+python scripts/probing/inspect_sample.py
+```
+
+Generate unique-count and random-20 decomposition reports:
+
+```bash
+python scripts/probing/analyze_and_decompose.py
+```
+
+Run native hard-filter audit:
+
+```bash
+python scripts/filtering/audit_native_traces.py --input-root data/Open-SWE-Traces --output-dir runs/native_trace_audit
+```
+
+Run exact Qwen token counting:
+
+```bash
+python scripts/probing/count_qwen_tokens_exact_parallel.py --input-root data/Open-SWE-Traces --output runs/native_trace_audit/qwen_exact_token_count.json --model Qwen/Qwen3-32B --workers 12
+```
+
+Build the balanced 5k SWIFT training probe:
+
+```bash
+python scripts/repurposing/build_swift_training_probe_5k.py
+```
+
+Build the balanced 500-row validation probe:
+
+```bash
+python scripts/repurposing/build_swift_validation_500.py
+```
+
+Try pi-mono-style conversion:
+
+```bash
+python scripts/repurposing/convert_openswe_to_pi_mono.py --input-root data/Open-SWE-Traces --output-root runs/pi_mono_converted
+```
+
+## Filtering Semantics
+
+The hard filter rejects trajectories with malformed tool-call JSON, tool-call/tool-result mismatch, polluted tool names or arguments, empty or inconsistent final patches, excessive length, repeated no-progress loops, and unresolved off-track traces.
+
+## Training Format Notes
+
+For SWIFT exports:
+
+- MiniMax trajectories are treated as thinking-mode data and `reasoning_content` is wrapped with `<think>...</think>`.
+- Qwen trajectories are treated as non-thinking-mode data.
+- `system`, `user`, and `tool` messages use `loss=false`; assistant messages use `loss=true`.
+