Organize Open-SWE-Traces data prep project

2026-06-24 22:34:04 +08:00
commit f06e573b04
27 changed files with 3264 additions and 0 deletions
--- a/docs/legacy_subproblem_decomposition_README.md
+++ b/docs/legacy_subproblem_decomposition_README.md
@@ -0,0 +1,241 @@
+# Open-SWE-Traces Subproblem Decomposition Probe
+
+## Dataset Counts
+
+Full dataset location:
+
+```text
+/ssd/workspace/yi/openswetraces_probe/Open-SWE-Traces
+```
+
+Probe outputs:
+
+```text
+/ssd/workspace/yi/openswetraces_probe/subproblem_decomposition
+```
+
+Global counts from lightweight parquet column scan:
+
+```text
+total trajectories:       207,489
+unique problems:           22,320
+unique repositories:        2,654
+```
+
+Per config:
+
+| config | thinking mode | scaffold | rows | unique problems | unique repos |
+|---|---:|---|---:|---:|---:|
+| minimax_m25_openhands | thinking | OpenHands | 49,948 | 20,098 | 2,606 |
+| minimax_m25_sweagent | thinking | SWE-agent | 57,268 | 20,791 | 2,582 |
+| qwen35_openhands | non-thinking | OpenHands | 55,488 | 20,362 | 2,589 |
+| qwen35_sweagent | non-thinking | SWE-agent | 44,785 | 18,211 | 2,529 |
+
+Problem coverage across the four configs:
+
+```text
+appears in 4 configs: 15,605 problems
+appears in 3 configs:  4,364 problems
+appears in 2 configs:  1,599 problems
+appears in 1 config:     752 problems
+```
+
+The data construction assumption is reflected in the probe: MiniMax rows are treated as thinking-mode trajectories; Qwen rows are treated as non-thinking trajectories.
+
+## Decomposition Logic
+
+The script `analyze_and_decompose.py` treats a trajectory as alternating assistant tool calls and tool observations. It assigns each assistant/tool pair to one of these phases:
+
+```text
+understand_task
+explore_repo
+locate_relevant_code
+inspect_code
+reproduce_or_probe
+edit_solution
+verify_solution
+cleanup
+review_diff
+submit
+other
+```
+
+The phase classifier is mostly rule-based:
+
+- `find`, `ls`, `tree`, `rg --files` -> repository exploration / code location.
+- `grep`, `rg`, symbol search -> locate relevant code.
+- `view`, `cat`, `sed -n`, `head`, `tail` -> inspect code.
+- `python repro`, `test_reproduce`, ad-hoc scripts -> reproduce or probe behavior.
+- `str_replace_editor create/insert/str_replace` -> edit solution.
+- `pytest`, `go test`, `npm test`, `cargo test`, `mvn test`, etc. -> verify solution.
+- `git diff`, `git status` -> review final patch.
+- `rm`, cleanup language, temporary file deletion -> cleanup.
+- `finish` / `submit` -> submit.
+
+It also extracts files, commands, short observations, and constructs candidate QA pairs for every segment.
+
+## Random-20 Probe Result
+
+Random seed:
+
+```text
+20260623
+```
+
+Files produced:
+
+```text
+unique_counts.json
+random20_decomposed.json
+report.json
+```
+
+Random 20 summary:
+
+```text
+items: 20
+usable_for_subproblem_training: 20 / 20
+avg_segments_per_trace: 58.8
+```
+
+Phase counts across the 20 traces:
+
+```text
+understand_task:       20
+explore_repo:          34
+locate_relevant_code: 143
+inspect_code:         211
+reproduce_or_probe:   317
+edit_solution:        151
+verify_solution:      161
+cleanup:               30
+review_diff:           80
+submit:                29
+```
+
+Interpretation: all 20 traces contain enough structure to create subproblem training data. The current segmentation is intentionally low-level and over-segments long agent loops. It is good for action-level supervision, but high-level SFT should merge adjacent phases into coarser task-oriented chunks.
+
+## Recommended Subproblem Formats
+
+### QA Format
+
+Use this when training repository-aware reasoning without replaying every tool result.
+
+```json
+{
+  "question": "In repo owner/name, how can we locate the code responsible for <issue>?",
+  "answer": {
+    "phase": "locate_relevant_code",
+    "files": ["..."],
+    "commands": ["rg ...", "find ..."],
+    "evidence": ["symbol X is defined in file Y"],
+    "next_step": "inspect_code"
+  }
+}
+```
+
+Best phases for QA:
+
+```text
+locate_relevant_code
+inspect_code
+reproduce_or_probe
+verify_solution
+review_diff
+```
+
+### Question-Trajectory Format
+
+Use this when training tool-use policy or pi-mono-like agent behavior.
+
+```json
+{
+  "question": "In repo owner/name, implement the edit that fixes <localized bug>.",
+  "trajectory": [
+    {"role": "assistant", "content": "...", "tool_calls": [...]},
+    {"role": "tool", "content": "..."}
+  ],
+  "target_patch": "diff --git ...",
+  "phase": "edit_solution"
+}
+```
+
+Best phases for Question-Trajectory:
+
+```text
+explore_repo -> locate_relevant_code -> inspect_code
+inspect_code -> edit_solution
+edit_solution -> verify_solution
+verify_solution -> review_diff -> submit
+```
+
+## Conversion Notes for pi-mono
+
+Tool dialect normalization is required:
+
+| Source tool | pi-mono target |
+|---|---|
+| `execute_bash`, `bash` | `bash` |
+| `str_replace_editor` `view` directory | `ls` |
+| `str_replace_editor` `view` file/range | `read` |
+| `str_replace_editor` `create` | `write` |
+| `str_replace_editor` `str_replace` / `insert` | `edit` |
+| `finish`, `submit` | final answer / stop |
+
+Path normalization is required:
+
+```text
+/testbed -> repo root
+/workspace/<repo>__1.0 -> repo root
+absolute file paths -> repo-relative paths where possible
+```
+
+For Qwen3.6 27B SFT:
+
+- Qwen traces should be used as non-thinking examples.
+- MiniMax traces should be used as thinking examples, with `reasoning_content` mapped to the thinking channel.
+- For long traces, prefer subproblem chunks over full-trajectory examples.
+- Use resolved=1 as the highest-quality subset, but unresolved traces can still train exploration, localization, and verification behavior if segmented before the bad final edit.
+
+## Next Improvements
+
+1. Add a second-pass merger that combines adjacent low-level segments into 6-10 high-level chunks per trace.
+2. Add semantic labels using issue text and touched files, e.g. `api_contract_update`, `test_regression_reproduction`, `parser_edge_case_fix`.
+3. Score each segment for trainability: contains objective, command/action, evidence, and outcome.
+4. Export two datasets:
+   - `subproblem_qa.jsonl`
+   - `subproblem_question_trajectory.jsonl`
+
+## Coarse Task-Category Decomposition
+
+A second script, `coarse_decompose.py`, aggregates fine segments into six coarse task categories per trace:
+
+```text
+understand
+locate
+diagnose
+fix
+verify
+finalize
+```
+
+For the random-20 sample, this reduced 1,176 fine segments to 120 coarse segments:
+
+```text
+avg fine segments per trace:   58.8
+avg coarse segments per trace:  6.0
+```
+
+This coarse output is better for SFT subproblem construction. Each coarse segment contains deduplicated files, commands, observations, assistant intents, and two training views:
+
+```text
+training_views.qa
+training_views.question_trajectory
+```
+
+Main output:
+
+```text
+random20_coarse_decomposed.json
+coarse_report.json
+```