OpenSWETraces_cleanup/README.md

SWE Data Processing

swe-data-processing is a conservative, auditable Python pipeline for cleaning nvidia/Open-SWE-Traces before supervised fine-tuning of smaller coding agents.

The project is designed for a restricted environment in which repository containers and agent sandboxes are unavailable. It combines deterministic local checks with GLM-5.2 API judgments. It does not claim that static inspection can prove code correctness. Instead, it separates internally consistent silver positives, useful negatives, safely repairable formatting issues, unverified records, and corrupted records.

Dataset snapshot

The current local snapshot contains 207,489 trajectories over 22,320 unique issues. The immutable dataset outcome field is named resolved:

resolved	Meaning	Count	Percentage
1	Successful candidate	65,244	31.44%
0	Explicit failure	95,487	46.02%
-1	Unknown outcome	46,758	22.54%

Records must be split by instance_id, not by trajectory, to prevent the same issue from leaking across train and evaluation sets.

Safety model

Static cleanup may improve representation quality, but it may not create new execution facts. The implementation enforces these invariants locally after every model response:

1. resolved is immutable.
2. A resolved=0 record can never become a full successful SFT example.
3. A resolved=-1 record cannot become SFT_FULL without execution.
4. Existing tool observations cannot be rewritten or fabricated.
5. Model and reference patches are immutable during static repair.
6. Truncated records have a maximum use of SFT_STEP_ONLY.
7. A silver positive requires reliable post-edit verification, no hard failures, and a passing value for every QC dimension.
8. Every repair requires a separate review API call.

The API model proposes decisions and repair plans. Deterministic Python code validates schemas, enforces outcome policy, applies only allowlisted edits, and records provenance.

Long trajectories are sent to the API through a turn-preserving evidence view. Test observations, state-changing turns, malformed calls, final turns, patches, and stable turn IDs receive priority. Shortened values include their original character count and SHA-256 hash. This compaction affects only the API prompt; classification never rewrites source JSONL or Parquet records.

API endpoint

The configured gateway is OpenAI chat-completions compatible:

POST https://llm-api.cowin.run/v1/chat/completions

/v1/text-completion and /v1/text-completions resolve to the gateway's web console rather than an inference API, so they are not used.

The default model is glm-5.2. The API key is read only from GLM_API_KEY. Do not write a key into source code, command history, output manifests, or this README.

Project layout

swe_data_processing/
├── pyproject.toml
├── README.md
├── changelog.md             # Versioned strategy changes
├── .env.example
├── src/swe_data_processing/
│   ├── cli.py                 # Command-line entry points
│   ├── client.py              # GLM API client and JSON validation
│   ├── config.py              # Environment-only runtime settings
│   ├── evidence.py            # Prompt-only evidence compaction
│   ├── features.py            # Deterministic static evidence extraction
│   ├── io.py                  # Streaming Parquet and JSONL readers
│   ├── policy.py              # Immutable local policy guards
│   ├── repair.py              # Allowlisted deterministic repair application
│   ├── resources.py           # Packaged prompt/schema loading
│   ├── workflow.py            # Classify, plan repair, and review stages
│   ├── prompts/               # Version-controlled GLM system prompts
│   └── schemas/               # JSON Schemas for every API stage
├── tests/                     # Offline unit tests; no network calls
├── scripts/                   # Existing sampling and profiling utilities
├── raw/Open-SWE-Traces/       # Downloaded dataset; ignored by Git
├── samples/                   # Human-review samples
├── reports/                   # Dataset reports and QC rubrics
└── qc_outputs/                # Generated manifests; ignored by Git

Installation

The existing remote virtual environment can install the package in editable mode:

cd /mnt/beegfs/yi/swe_data_processing
./.venv/bin/pip install -e '.[dev]'

For a fresh environment:

python3 -m venv .venv
./.venv/bin/pip install --upgrade pip
./.venv/bin/pip install -e '.[dev]'

Configuration

Export credentials in the shell that launches the pipeline:

export GLM_API_KEY='your-runtime-secret'
export GLM_API_BASE='https://llm-api.cowin.run'
export GLM_API_PATH='/v1/chat/completions'
export GLM_MODEL='glm-5.2'

Optional settings and their defaults:

export GLM_TIMEOUT_SECONDS=300
export GLM_MAX_RETRIES=5
export GLM_MAX_TOKENS=8192
export GLM_TEMPERATURE=0.0
export GLM_REASONING_EFFORT=high
export GLM_THINKING_ENABLED=true

GLM_MAX_RETRIES=5 means one initial request plus at most five retries. The client retries timeouts, connection failures, HTTP 429/5xx responses, malformed JSON, and schema-invalid model output with bounded exponential backoff. HTTP 401/403 authentication failures are never retried.

If the gateway rejects GLM-specific thinking or reasoning_effort fields, the client automatically retries with the portable OpenAI-compatible request subset.

Commands

1. Verify API authentication and structured output

swe-qc smoke-test

The command prints the endpoint, model, request ID, usage, and a tiny validated JSON response. It never prints the API key.

2. Extract deterministic evidence without API calls

From the 20-record review sample:

mkdir -p qc_outputs
swe-qc features \
  --input samples/sample_20_seed_20260805.jsonl \
  --output qc_outputs/sample20.features.jsonl \
  --errors qc_outputs/sample20.features.errors.jsonl

From all Parquet shards:

swe-qc features \
  --input raw/Open-SWE-Traces \
  --output qc_outputs/all.features.jsonl \
  --errors qc_outputs/all.features.errors.jsonl \
  --resume

Features include malformed tool arguments, unknown tools, role alternation, state-changing turns, post-edit test evidence, masked shell pipelines, patch file sets, patch size ratios, and explicit user constraints.

3. Classify trajectories through GLM-5.2

Run a small pilot first:

swe-qc classify \
  --input samples/sample_20_seed_20260805.jsonl \
  --output qc_outputs/sample20.classifications.jsonl \
  --errors qc_outputs/sample20.classification.errors.jsonl \
  --limit 20 \
  --resume

The classifier returns one of:

• ACCEPT_SILVER_POSITIVE
• ACCEPT_NEGATIVE
• STATIC_REPAIR
• HOLD_UNVERIFIED
• REJECT

Training use is tracked separately as SFT_FULL, SFT_STEP_ONLY, DPO_REJECTED, ERROR_ANALYSIS, HOLD, or DROP.

4. Locate a safe prefix and score it independently

audit uses two isolated GLM calls for failed and unknown trajectories:

1. The boundary call receives the trajectory as roughly 32-turn blocks without splitting an assistant action from its immediate tool result. It cannot see resolved, model patches, or reference patches. It selects the earliest unrepaired major/critical assistant turn, keeps a safe full process trace, or returns HOLD when neither decision is supported.
2. Python slices the exact messages before that assistant turn. The quality call receives only this materialized prefix, so suffix behavior cannot affect its score.

Successful trajectories skip boundary selection and are scored as complete trajectories. The command remains simple:

swe-qc audit \
  --input samples/sample_20_seed_20260805.jsonl \
  --output qc_outputs/sample20.audits.jsonl \
  --errors qc_outputs/sample20.audit.errors.jsonl \
  --workers 5 \
  --resume

--workers bounds the number of records processed concurrently. JSONL writes remain serialized in the main thread, so each completed record is appended atomically even when API requests run in parallel. Output order follows request completion order; sample_id remains the stable join key.

The gateway previously returned frequent HTTP 429 responses at 20 workers, so five workers is the practical default for long runs unless the service limit is raised.

The quality call lists concrete erroneous and inefficient assistant actions and scores five 0-20 dimensions: planning, investigation, tool use and observation, progress, and clarity/efficiency. Python recomputes issue counts, sums the five dimensions, and assigns:

• HIGH: 80-100
• MEDIUM: 60-79
• LOW: 40-59
• REJECT: below 40, or an unrepaired major/critical problem remains

Incomplete prefixes can still be useful. Normal failed experiments are retained when the agent later diagnoses, repairs, and verifies them. A prefix ending in a complete tool result is valid process-SFT structure.

5. Create static repair plans

Only classifications that explicitly return STATIC_REPAIR are processed by default:

swe-qc repair-plan \
  --input samples/sample_20_seed_20260805.jsonl \
  --classifications qc_outputs/sample20.classifications.jsonl \
  --output qc_outputs/sample20.repair-plans.jsonl \
  --errors qc_outputs/sample20.repair-plan.errors.jsonl \
  --resume

The model may propose only operations from the allowlist. Code changes, test result synthesis, patch replacement, and outcome upgrades are prohibited.

6. Apply plans deterministically

swe-qc apply-repair \
  --input samples/sample_20_seed_20260805.jsonl \
  --plans qc_outputs/sample20.repair-plans.jsonl \
  --output qc_outputs/sample20.repaired.jsonl \
  --errors qc_outputs/sample20.apply.errors.jsonl \
  --resume

The applier checks target turns, read-only pair deletion, immutable patch fields, and retained tool-output hashes. It emits a structured diff with input and output SHA-256 hashes.

7. Independently review repairs

swe-qc review \
  --input samples/sample_20_seed_20260805.jsonl \
  --classifications qc_outputs/sample20.classifications.jsonl \
  --plans qc_outputs/sample20.repair-plans.jsonl \
  --repaired qc_outputs/sample20.repaired.jsonl \
  --output qc_outputs/sample20.reviews.jsonl \
  --errors qc_outputs/sample20.review.errors.jsonl \
  --resume

Approval means that a static repair preserved evidence and structure. It does not mean that the code patch was executed or proved correct.

Output and resume behavior

Pipeline manifests are append-only JSONL. --resume reads completed sample IDs from the output and skips them. Each API response includes non-sensitive provenance:

• model and endpoint;
• gateway request ID;
• token usage when available;
• input SHA-256;
• compatibility-fallback flag;
• UTC creation time.

Errors are written as compact records containing sample ID, stage, exception type, and a bounded message. Raw prompts and credentials are not copied into error logs.

Recommended rollout

1. Run offline tests.
2. Run smoke-test once.
3. Run classification on the 20 manually reviewed records.
4. Compare GLM decisions with the human labels.
5. Build a 2,000-record stratified calibration set.
6. Require at least 95% precision for ACCEPT_SILVER_POSITIVE before scaling.
7. Keep resolved=0 as negative data rather than attempting to turn it into positive trajectories.
8. Keep resolved=-1 in an unverified manifest unless it contains explicit, reliable failure evidence.

Development

Run all offline tests:

./.venv/bin/pytest

Run lint checks:

./.venv/bin/ruff check src tests

Tests use an httpx.MockTransport and never contact the GLM endpoint.

Credential handling

• .env files are ignored by Git.
• The provided API key is not stored anywhere in this project.
• Use a secret manager or a protected runtime environment variable for batch jobs.
• Rotate the key if it has been copied into a public log, issue, or repository.