OpenSWETraces_cleanup/README.md

# SWE Data Processing

`swe-data-processing` is a conservative, auditable Python pipeline for cleaning
[`nvidia/Open-SWE-Traces`](https://huggingface.co/datasets/nvidia/Open-SWE-Traces)
before supervised fine-tuning of smaller coding agents.

The project is designed for a restricted environment in which repository
containers and agent sandboxes are unavailable. It combines deterministic local
checks with GLM-5.2 API judgments. It does **not** claim that static inspection can
prove code correctness. Instead, it separates internally consistent silver
positives, useful negatives, safely repairable formatting issues, unverified
records, and corrupted records.

## Dataset snapshot

The current local snapshot contains 207,489 trajectories over 22,320 unique
issues. The immutable dataset outcome field is named `resolved`:

| `resolved` | Meaning | Count | Percentage |
|---:|---|---:|---:|
| `1` | Successful candidate | 65,244 | 31.44% |
| `0` | Explicit failure | 95,487 | 46.02% |
| `-1` | Unknown outcome | 46,758 | 22.54% |

Records must be split by `instance_id`, not by trajectory, to prevent the same
issue from leaking across train and evaluation sets.

## Safety model

Static cleanup may improve representation quality, but it may not create new
execution facts. The implementation enforces these invariants locally after
every model response:

1. `resolved` is immutable.
2. A `resolved=0` record can never become a full successful SFT example.
3. A `resolved=-1` record cannot become `SFT_FULL` without execution.
4. Existing tool observations cannot be rewritten or fabricated.
5. Model and reference patches are immutable during static repair.
6. Truncated records have a maximum use of `SFT_STEP_ONLY`.
7. A silver positive requires reliable post-edit verification, no hard failures,
   and a passing value for every QC dimension.
8. Every repair requires a separate review API call.

The API model proposes decisions and repair plans. Deterministic Python code
validates schemas, enforces outcome policy, applies only allowlisted edits, and
records provenance.

Long trajectories are sent to the API through a turn-preserving evidence view.
Test observations, state-changing turns, malformed calls, final turns, patches,
and stable turn IDs receive priority. Shortened values include their original
character count and SHA-256 hash. This compaction affects only the API prompt;
classification never rewrites source JSONL or Parquet records.

## API endpoint

The configured gateway is OpenAI chat-completions compatible:

```text
POST https://llm-api.cowin.run/v1/chat/completions
```

`/v1/text-completion` and `/v1/text-completions` resolve to the gateway's web
console rather than an inference API, so they are not used.

The default model is `glm-5.2`. The API key is read only from `GLM_API_KEY`. Do
not write a key into source code, command history, output manifests, or this
README.

## Project layout

```text
swe_data_processing/
├── pyproject.toml
├── README.md
├── .env.example
├── src/swe_data_processing/
│   ├── cli.py                 # Command-line entry points
│   ├── client.py              # GLM API client and JSON validation
│   ├── config.py              # Environment-only runtime settings
│   ├── evidence.py            # Prompt-only evidence compaction
│   ├── features.py            # Deterministic static evidence extraction
│   ├── io.py                  # Streaming Parquet and JSONL readers
│   ├── policy.py              # Immutable local policy guards
│   ├── repair.py              # Allowlisted deterministic repair application
│   ├── resources.py           # Packaged prompt/schema loading
│   ├── workflow.py            # Classify, plan repair, and review stages
│   ├── prompts/               # Version-controlled GLM system prompts
│   └── schemas/               # JSON Schemas for every API stage
├── tests/                     # Offline unit tests; no network calls
├── scripts/                   # Existing sampling and profiling utilities
├── raw/Open-SWE-Traces/       # Downloaded dataset; ignored by Git
├── samples/                   # Human-review samples
├── reports/                   # Dataset reports and QC rubrics
└── qc_outputs/                # Generated manifests; ignored by Git
```

## Installation

The existing remote virtual environment can install the package in editable
mode:

```bash
cd /mnt/beegfs/yi/swe_data_processing
./.venv/bin/pip install -e '.[dev]'
```

For a fresh environment:

```bash
python3 -m venv .venv
./.venv/bin/pip install --upgrade pip
./.venv/bin/pip install -e '.[dev]'
```

## Configuration

Export credentials in the shell that launches the pipeline:

```bash
export GLM_API_KEY='your-runtime-secret'
export GLM_API_BASE='https://llm-api.cowin.run'
export GLM_API_PATH='/v1/chat/completions'
export GLM_MODEL='glm-5.2'
```

Optional settings and their defaults:

```bash
export GLM_TIMEOUT_SECONDS=300
export GLM_MAX_RETRIES=5
export GLM_MAX_TOKENS=8192
export GLM_TEMPERATURE=0.0
export GLM_REASONING_EFFORT=high
export GLM_THINKING_ENABLED=true
```

`GLM_MAX_RETRIES=5` means one initial request plus at most five retries. The
client retries timeouts, connection failures, HTTP 429/5xx responses, malformed
JSON, and schema-invalid model output with bounded exponential backoff. HTTP
401/403 authentication failures are never retried.

If the gateway rejects GLM-specific `thinking` or `reasoning_effort` fields, the
client automatically retries with the portable OpenAI-compatible request subset.

## Commands

### 1. Verify API authentication and structured output

```bash
swe-qc smoke-test
```

The command prints the endpoint, model, request ID, usage, and a tiny validated
JSON response. It never prints the API key.

### 2. Extract deterministic evidence without API calls

From the 20-record review sample:

```bash
mkdir -p qc_outputs
swe-qc features \
  --input samples/sample_20_seed_20260805.jsonl \
  --output qc_outputs/sample20.features.jsonl \
  --errors qc_outputs/sample20.features.errors.jsonl
```

From all Parquet shards:

```bash
swe-qc features \
  --input raw/Open-SWE-Traces \
  --output qc_outputs/all.features.jsonl \
  --errors qc_outputs/all.features.errors.jsonl \
  --resume
```

Features include malformed tool arguments, unknown tools, role alternation,
state-changing turns, post-edit test evidence, masked shell pipelines, patch file
sets, patch size ratios, and explicit user constraints.

### 3. Classify trajectories through GLM-5.2

Run a small pilot first:

```bash
swe-qc classify \
  --input samples/sample_20_seed_20260805.jsonl \
  --output qc_outputs/sample20.classifications.jsonl \
  --errors qc_outputs/sample20.classification.errors.jsonl \
  --limit 20 \
  --resume
```

The classifier returns one of:

- `ACCEPT_SILVER_POSITIVE`
- `ACCEPT_NEGATIVE`
- `STATIC_REPAIR`
- `HOLD_UNVERIFIED`
- `REJECT`

Training use is tracked separately as `SFT_FULL`, `SFT_STEP_ONLY`,
`DPO_REJECTED`, `ERROR_ANALYSIS`, `HOLD`, or `DROP`.

### 4. Audit educational process quality

This stage does not repair trajectories. It identifies defensible erroneous or
inefficient calls, scores six process-quality dimensions, and optionally finds
a causal first-bad assistant turn for prefix-only learning:

```bash
swe-qc audit \
  --input samples/sample_20_seed_20260805.jsonl \
  --output qc_outputs/sample20.audits.jsonl \
  --errors qc_outputs/sample20.audit.errors.jsonl \
  --workers 20 \
  --resume
```

`--workers` bounds the number of records processed concurrently. JSONL writes
remain serialized in the main thread, so each completed record is appended
atomically even when API requests run in parallel. Output order follows request
completion order; `sample_id` remains the stable join key.

The deterministic score combines weighted process dimensions with penalties
for minor, major, and critical behavior issues. Failed exploratory calls are
not penalized when the agent interprets them correctly and recovers.

The causal audit payload excludes the reference patch and every derived signal,
including reference file names, patch length, and model/reference size ratios.
The immutable outcome is used only to select the mode: failed or unknown
trajectories can become process-prefix candidates, but never full-trajectory
SFT candidates.

### 5. Create static repair plans

Only classifications that explicitly return `STATIC_REPAIR` are processed by
default:

```bash
swe-qc repair-plan \
  --input samples/sample_20_seed_20260805.jsonl \
  --classifications qc_outputs/sample20.classifications.jsonl \
  --output qc_outputs/sample20.repair-plans.jsonl \
  --errors qc_outputs/sample20.repair-plan.errors.jsonl \
  --resume
```

The model may propose only operations from the allowlist. Code changes, test
result synthesis, patch replacement, and outcome upgrades are prohibited.

### 6. Apply plans deterministically

```bash
swe-qc apply-repair \
  --input samples/sample_20_seed_20260805.jsonl \
  --plans qc_outputs/sample20.repair-plans.jsonl \
  --output qc_outputs/sample20.repaired.jsonl \
  --errors qc_outputs/sample20.apply.errors.jsonl \
  --resume
```

The applier checks target turns, read-only pair deletion, immutable patch fields,
and retained tool-output hashes. It emits a structured diff with input and output
SHA-256 hashes.

### 7. Independently review repairs

```bash
swe-qc review \
  --input samples/sample_20_seed_20260805.jsonl \
  --classifications qc_outputs/sample20.classifications.jsonl \
  --plans qc_outputs/sample20.repair-plans.jsonl \
  --repaired qc_outputs/sample20.repaired.jsonl \
  --output qc_outputs/sample20.reviews.jsonl \
  --errors qc_outputs/sample20.review.errors.jsonl \
  --resume
```

Approval means that a static repair preserved evidence and structure. It does not
mean that the code patch was executed or proved correct.

## Output and resume behavior

Pipeline manifests are append-only JSONL. `--resume` reads completed sample IDs
from the output and skips them. Each API response includes non-sensitive
provenance:

- model and endpoint;
- gateway request ID;
- token usage when available;
- input SHA-256;
- compatibility-fallback flag;
- UTC creation time.

Errors are written as compact records containing sample ID, stage, exception
type, and a bounded message. Raw prompts and credentials are not copied into
error logs.

## Recommended rollout

1. Run offline tests.
2. Run `smoke-test` once.
3. Run classification on the 20 manually reviewed records.
4. Compare GLM decisions with the human labels.
5. Build a 2,000-record stratified calibration set.
6. Require at least 95% precision for `ACCEPT_SILVER_POSITIVE` before scaling.
7. Keep `resolved=0` as negative data rather than attempting to turn it into
   positive trajectories.
8. Keep `resolved=-1` in an unverified manifest unless it contains explicit,
   reliable failure evidence.

## Development

Run all offline tests:

```bash
./.venv/bin/pytest
```

Run lint checks:

```bash
./.venv/bin/ruff check src tests
```

Tests use an `httpx.MockTransport` and never contact the GLM endpoint.

## Credential handling

- `.env` files are ignored by Git.
- The provided API key is not stored anywhere in this project.
- Use a secret manager or a protected runtime environment variable for batch
  jobs.
- Rotate the key if it has been copied into a public log, issue, or repository.