Initial Open-SWE-Traces cleanup pipeline

2026-08-06 22:53:47 +08:00
commit 044bd03f0e
35 changed files with 3638 additions and 0 deletions
@@ -0,0 +1,325 @@
+# SWE Data Processing
+
+`swe-data-processing` is a conservative, auditable Python pipeline for cleaning
+[`nvidia/Open-SWE-Traces`](https://huggingface.co/datasets/nvidia/Open-SWE-Traces)
+before supervised fine-tuning of smaller coding agents.
+
+The project is designed for a restricted environment in which repository
+containers and agent sandboxes are unavailable. It combines deterministic local
+checks with GLM-5.2 API judgments. It does **not** claim that static inspection can
+prove code correctness. Instead, it separates internally consistent silver
+positives, useful negatives, safely repairable formatting issues, unverified
+records, and corrupted records.
+
+## Dataset snapshot
+
+The current local snapshot contains 207,489 trajectories over 22,320 unique
+issues. The immutable dataset outcome field is named `resolved`:
+
+| `resolved` | Meaning | Count | Percentage |
+|---:|---|---:|---:|
+| `1` | Successful candidate | 65,244 | 31.44% |
+| `0` | Explicit failure | 95,487 | 46.02% |
+| `-1` | Unknown outcome | 46,758 | 22.54% |
+
+Records must be split by `instance_id`, not by trajectory, to prevent the same
+issue from leaking across train and evaluation sets.
+
+## Safety model
+
+Static cleanup may improve representation quality, but it may not create new
+execution facts. The implementation enforces these invariants locally after
+every model response:
+
+1. `resolved` is immutable.
+2. A `resolved=0` record can never become a full successful SFT example.
+3. A `resolved=-1` record cannot become `SFT_FULL` without execution.
+4. Existing tool observations cannot be rewritten or fabricated.
+5. Model and reference patches are immutable during static repair.
+6. Truncated records have a maximum use of `SFT_STEP_ONLY`.
+7. A silver positive requires reliable post-edit verification, no hard failures,
+   and a passing value for every QC dimension.
+8. Every repair requires a separate review API call.
+
+The API model proposes decisions and repair plans. Deterministic Python code
+validates schemas, enforces outcome policy, applies only allowlisted edits, and
+records provenance.
+
+Long trajectories are sent to the API through a turn-preserving evidence view.
+Test observations, state-changing turns, malformed calls, final turns, patches,
+and stable turn IDs receive priority. Shortened values include their original
+character count and SHA-256 hash. This compaction affects only the API prompt;
+classification never rewrites source JSONL or Parquet records.
+
+## API endpoint
+
+The configured gateway is OpenAI chat-completions compatible:
+
+```text
+POST https://llm-api.cowin.run/v1/chat/completions
+```
+
+`/v1/text-completion` and `/v1/text-completions` resolve to the gateway's web
+console rather than an inference API, so they are not used.
+
+The default model is `glm-5.2`. The API key is read only from `GLM_API_KEY`. Do
+not write a key into source code, command history, output manifests, or this
+README.
+
+## Project layout
+
+```text
+swe_data_processing/
+├── pyproject.toml
+├── README.md
+├── .env.example
+├── src/swe_data_processing/
+│   ├── cli.py                 # Command-line entry points
+│   ├── client.py              # GLM API client and JSON validation
+│   ├── config.py              # Environment-only runtime settings
+│   ├── evidence.py            # Prompt-only evidence compaction
+│   ├── features.py            # Deterministic static evidence extraction
+│   ├── io.py                  # Streaming Parquet and JSONL readers
+│   ├── policy.py              # Immutable local policy guards
+│   ├── repair.py              # Allowlisted deterministic repair application
+│   ├── resources.py           # Packaged prompt/schema loading
+│   ├── workflow.py            # Classify, plan repair, and review stages
+│   ├── prompts/               # Version-controlled GLM system prompts
+│   └── schemas/               # JSON Schemas for every API stage
+├── tests/                     # Offline unit tests; no network calls
+├── scripts/                   # Existing sampling and profiling utilities
+├── raw/Open-SWE-Traces/       # Downloaded dataset; ignored by Git
+├── samples/                   # Human-review samples
+├── reports/                   # Dataset reports and QC rubrics
+└── qc_outputs/                # Generated manifests; ignored by Git
+```
+
+## Installation
+
+The existing remote virtual environment can install the package in editable
+mode:
+
+```bash
+cd /mnt/beegfs/yi/swe_data_processing
+./.venv/bin/pip install -e '.[dev]'
+```
+
+For a fresh environment:
+
+```bash
+python3 -m venv .venv
+./.venv/bin/pip install --upgrade pip
+./.venv/bin/pip install -e '.[dev]'
+```
+
+## Configuration
+
+Export credentials in the shell that launches the pipeline:
+
+```bash
+export GLM_API_KEY='your-runtime-secret'
+export GLM_API_BASE='https://llm-api.cowin.run'
+export GLM_API_PATH='/v1/chat/completions'
+export GLM_MODEL='glm-5.2'
+```
+
+Optional settings and their defaults:
+
+```bash
+export GLM_TIMEOUT_SECONDS=300
+export GLM_MAX_RETRIES=3
+export GLM_MAX_TOKENS=8192
+export GLM_TEMPERATURE=0.0
+export GLM_REASONING_EFFORT=high
+export GLM_THINKING_ENABLED=true
+```
+
+If the gateway rejects GLM-specific `thinking` or `reasoning_effort` fields, the
+client automatically retries with the portable OpenAI-compatible request subset.
+
+## Commands
+
+### 1. Verify API authentication and structured output
+
+```bash
+swe-qc smoke-test
+```
+
+The command prints the endpoint, model, request ID, usage, and a tiny validated
+JSON response. It never prints the API key.
+
+### 2. Extract deterministic evidence without API calls
+
+From the 20-record review sample:
+
+```bash
+mkdir -p qc_outputs
+swe-qc features \
+  --input samples/sample_20_seed_20260805.jsonl \
+  --output qc_outputs/sample20.features.jsonl \
+  --errors qc_outputs/sample20.features.errors.jsonl
+```
+
+From all Parquet shards:
+
+```bash
+swe-qc features \
+  --input raw/Open-SWE-Traces \
+  --output qc_outputs/all.features.jsonl \
+  --errors qc_outputs/all.features.errors.jsonl \
+  --resume
+```
+
+Features include malformed tool arguments, unknown tools, role alternation,
+state-changing turns, post-edit test evidence, masked shell pipelines, patch file
+sets, patch size ratios, and explicit user constraints.
+
+### 3. Classify trajectories through GLM-5.2
+
+Run a small pilot first:
+
+```bash
+swe-qc classify \
+  --input samples/sample_20_seed_20260805.jsonl \
+  --output qc_outputs/sample20.classifications.jsonl \
+  --errors qc_outputs/sample20.classification.errors.jsonl \
+  --limit 20 \
+  --resume
+```
+
+The classifier returns one of:
+
+- `ACCEPT_SILVER_POSITIVE`
+- `ACCEPT_NEGATIVE`
+- `STATIC_REPAIR`
+- `HOLD_UNVERIFIED`
+- `REJECT`
+
+Training use is tracked separately as `SFT_FULL`, `SFT_STEP_ONLY`,
+`DPO_REJECTED`, `ERROR_ANALYSIS`, `HOLD`, or `DROP`.
+
+### 4. Audit educational process quality
+
+This stage does not repair trajectories. It identifies defensible erroneous or
+inefficient calls, scores six process-quality dimensions, and optionally finds
+a causal first-bad assistant turn for prefix-only learning:
+
+```bash
+swe-qc audit \
+  --input samples/sample_20_seed_20260805.jsonl \
+  --output qc_outputs/sample20.audits.jsonl \
+  --errors qc_outputs/sample20.audit.errors.jsonl \
+  --resume
+```
+
+The deterministic score combines weighted process dimensions with penalties
+for minor, major, and critical behavior issues. Failed exploratory calls are
+not penalized when the agent interprets them correctly and recovers.
+
+The causal audit payload excludes the reference patch and every derived signal,
+including reference file names, patch length, and model/reference size ratios.
+The immutable outcome is used only to select the mode: failed or unknown
+trajectories can become process-prefix candidates, but never full-trajectory
+SFT candidates.
+
+### 5. Create static repair plans
+
+Only classifications that explicitly return `STATIC_REPAIR` are processed by
+default:
+
+```bash
+swe-qc repair-plan \
+  --input samples/sample_20_seed_20260805.jsonl \
+  --classifications qc_outputs/sample20.classifications.jsonl \
+  --output qc_outputs/sample20.repair-plans.jsonl \
+  --errors qc_outputs/sample20.repair-plan.errors.jsonl \
+  --resume
+```
+
+The model may propose only operations from the allowlist. Code changes, test
+result synthesis, patch replacement, and outcome upgrades are prohibited.
+
+### 6. Apply plans deterministically
+
+```bash
+swe-qc apply-repair \
+  --input samples/sample_20_seed_20260805.jsonl \
+  --plans qc_outputs/sample20.repair-plans.jsonl \
+  --output qc_outputs/sample20.repaired.jsonl \
+  --errors qc_outputs/sample20.apply.errors.jsonl \
+  --resume
+```
+
+The applier checks target turns, read-only pair deletion, immutable patch fields,
+and retained tool-output hashes. It emits a structured diff with input and output
+SHA-256 hashes.
+
+### 7. Independently review repairs
+
+```bash
+swe-qc review \
+  --input samples/sample_20_seed_20260805.jsonl \
+  --classifications qc_outputs/sample20.classifications.jsonl \
+  --plans qc_outputs/sample20.repair-plans.jsonl \
+  --repaired qc_outputs/sample20.repaired.jsonl \
+  --output qc_outputs/sample20.reviews.jsonl \
+  --errors qc_outputs/sample20.review.errors.jsonl \
+  --resume
+```
+
+Approval means that a static repair preserved evidence and structure. It does not
+mean that the code patch was executed or proved correct.
+
+## Output and resume behavior
+
+Pipeline manifests are append-only JSONL. `--resume` reads completed sample IDs
+from the output and skips them. Each API response includes non-sensitive
+provenance:
+
+- model and endpoint;
+- gateway request ID;
+- token usage when available;
+- input SHA-256;
+- compatibility-fallback flag;
+- UTC creation time.
+
+Errors are written as compact records containing sample ID, stage, exception
+type, and a bounded message. Raw prompts and credentials are not copied into
+error logs.
+
+## Recommended rollout
+
+1. Run offline tests.
+2. Run `smoke-test` once.
+3. Run classification on the 20 manually reviewed records.
+4. Compare GLM decisions with the human labels.
+5. Build a 2,000-record stratified calibration set.
+6. Require at least 95% precision for `ACCEPT_SILVER_POSITIVE` before scaling.
+7. Keep `resolved=0` as negative data rather than attempting to turn it into
+   positive trajectories.
+8. Keep `resolved=-1` in an unverified manifest unless it contains explicit,
+   reliable failure evidence.
+
+## Development
+
+Run all offline tests:
+
+```bash
+./.venv/bin/pytest
+```
+
+Run lint checks:
+
+```bash
+./.venv/bin/ruff check src tests
+```
+
+Tests use an `httpx.MockTransport` and never contact the GLM endpoint.
+
+## Credential handling
+
+- `.env` files are ignored by Git.
+- The provided API key is not stored anywhere in this project.
+- Use a secret manager or a protected runtime environment variable for batch
+  jobs.
+- Rotate the key if it has been copied into a public log, issue, or repository.