# TI Coding Agent Data Prep Skill

Use this repo for Open-SWE-Traces data preparation before coding-agent SFT experiments.

## Purpose

This project organizes the existing Open-SWE-Traces probing, native-trace filtering, and trajectory repurposing scripts into a Python project. It is intended for work on `nvidia/Open-SWE-Traces`, especially:

- inspecting dataset structure and source splits,
- auditing trajectories for SFT-quality hard filters,
- measuring Qwen tokenizer/token statistics,
- decomposing trajectories into task-oriented phases,
- exporting balanced ModelScope-SWIFT training/validation probes,
- experimenting with pi-mono-style conversion.

## Repository Layout

- `scripts/probing/`: dataset inspection, unique-count reports, tokenizer comparison, and token counting.
- `scripts/filtering/`: native trajectory audit and hard filtering.
- `scripts/repurposing/`: subproblem decomposition, coarse task-stage conversion, SWIFT export, and pi-mono conversion.
- `src/ti_coding_agent_data_prep/openswe/`: shared constants and path helpers.
- `data/`: local datasets; ignored by git.
- `runs/`: script outputs; ignored by git.
- `docs/`: retained legacy notes from the original probe workspace.

## Expected Data Location

By default scripts read:

```text
data/Open-SWE-Traces
```

On B300, prefer a symlink instead of copying the dataset:

```bash
ln -s /ssd/workspace/yi/openswetraces_probe/Open-SWE-Traces data/Open-SWE-Traces
```

## Environment

Use a repo-local environment. On B300, set proxy variables before installing or downloading:

```bash
export http_proxy=http://100.72.0.101:8888
export https_proxy=http://100.72.0.101:8888
export HTTP_PROXY=http://100.72.0.101:8888
export HTTPS_PROXY=http://100.72.0.101:8888
export HF_ENDPOINT=https://hf-mirror.com
uv venv .venv --python 3.10
source .venv/bin/activate
uv pip install -e '.[dev]'
```

## Main Entrypoints

Probe one sample per split:

```bash
python scripts/probing/inspect_sample.py
```

Generate unique-count and random-20 decomposition reports:

```bash
python scripts/probing/analyze_and_decompose.py
```

Run native hard-filter audit:

```bash
python scripts/filtering/audit_native_traces.py --input-root data/Open-SWE-Traces --output-dir runs/native_trace_audit
```

Run exact Qwen token counting:

```bash
python scripts/probing/count_qwen_tokens_exact_parallel.py --input-root data/Open-SWE-Traces --output runs/native_trace_audit/qwen_exact_token_count.json --model Qwen/Qwen3-32B --workers 12
```

Build the balanced 5k SWIFT training probe:

```bash
python scripts/repurposing/build_swift_training_probe_5k.py
```

Build the balanced 500-row validation probe:

```bash
python scripts/repurposing/build_swift_validation_500.py
```

Build the full hard-filter-kept SWIFT training set, approximately 190k rows with the current audit policy:

```bash
python scripts/repurposing/build_swift_full_kept.py --input-root data/Open-SWE-Traces --output-dir runs/training_full_kept_swift
```

Use `build_swift_full_kept.py` for real full-data SFT preparation. Use
`build_swift_training_probe_5k.py` plus `build_swift_validation_500.py` only for small probe runs and pipeline tests.

Try pi-mono-style conversion:

```bash
python scripts/repurposing/convert_openswe_to_pi_mono.py --input-root data/Open-SWE-Traces --output-root runs/pi_mono_converted
```

## Filtering Semantics

The hard filter rejects trajectories with malformed tool-call JSON, tool-call/tool-result mismatch, polluted tool names or arguments, empty or inconsistent final patches, excessive length, repeated no-progress loops, and unresolved off-track traces.

## Training Format Notes

For SWIFT exports:

- MiniMax trajectories are treated as thinking-mode data and `reasoning_content` is wrapped with `<think>...</think>`.
- Qwen trajectories are treated as non-thinking-mode data.
- `system`, `user`, and `tool` messages use `loss=false`; assistant messages use `loss=true`.