yi_lu/OpenSWETraces_cleanup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace LLM cleanup with deterministic profiling

Browse Source
This commit is contained in:
jiachun
2026-08-18 17:36:36 +08:00
parent d48cce3f81
commit 4a062134d3
47 changed files with 1371 additions and 3846 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+106 -331
View File
@@ -1,364 +1,139 @@
# SWE Data Processing
# Open-SWE-Traces Deterministic Profiler
`swe-data-processing` is a conservative, auditable Python pipeline for cleaning
[`nvidia/Open-SWE-Traces`](https://huggingface.co/datasets/nvidia/Open-SWE-Traces)
before supervised fine-tuning of smaller coding agents.
This project profiles `nvidia/Open-SWE-Traces` for coding-agent SFT without an
LLM judge, repository sandbox, or trajectory rewriting. It never claims that a
patch is correct. It records facts that can be reproduced from the dataset and
uses a small set of explicit heuristics to remove traces that are definitely
broken or unsuitable for a chosen training context.
The project is designed for a restricted environment in which repository
containers and agent sandboxes are unavailable. It combines deterministic local
checks with GLM-5.2 API judgments. It does **not** claim that static inspection can
prove code correctness. Instead, it separates internally consistent silver
positives, useful negatives, safely repairable formatting issues, unverified
records, and corrupted records.
The local dataset snapshot contains 207,489 trajectories over 22,320 issues:
## Dataset snapshot
| `resolved` | Meaning | Count |
|---:|---|---:|
| `1` | Externally marked successful | 65,244 |
| `0` | Externally marked failed | 95,487 |
| `-1` | Outcome unknown | 46,758 |
The current local snapshot contains 207,489 trajectories over 22,320 unique
issues. The immutable dataset outcome field is named `resolved`:
`resolved` is never changed. Only `resolved=1` is eligible for successful SFT;
the profiler can still describe failed and unknown traces for analysis. Final
decisions label `resolved=0` as `EXCLUDE_FAILED_OUTCOME` and `resolved=-1` as
`HOLD_UNVERIFIED_OUTCOME` rather than recommending them for training.
| `resolved` | Meaning | Count | Percentage |
|---:|---|---:|---:|
| `1` | Successful candidate | 65,244 | 31.44% |
| `0` | Explicit failure | 95,487 | 46.02% |
| `-1` | Unknown outcome | 46,758 | 22.54% |
## Design
Records must be split by `instance_id`, not by trajectory, to prevent the same
issue from leaking across train and evaluation sets.
The pipeline has three commands:
## Safety model
1. `profile` streams the original JSONL or Parquet dataset and writes one
deterministic metrics row per trajectory.
2. `summarize` computes dataset quantiles and writes transparent decisions.
3. `sample` selects full trajectories from every rule and length bucket for
human validation.
Static cleanup may improve representation quality, but it may not create new
execution facts. The implementation enforces these invariants locally after
every model response:
No command edits, truncates, repairs, or invents trajectory turns. Source data
and generated files are joined by `trajectory_id`/`sample_id`.
1. `resolved` is immutable.
2. A `resolved=0` record can never become a full successful SFT example.
3. A `resolved=-1` record cannot become `SFT_FULL` without execution.
4. Existing tool observations cannot be rewritten or fabricated.
5. Model and reference patches are immutable during static repair.
6. Truncated records have a maximum use of `SFT_STEP_ONLY`.
7. A silver positive requires reliable post-edit verification, no hard failures,
and a passing value for every QC dimension.
8. Every repair requires a separate review API call.
## Metrics
The API model proposes decisions and repair plans. Deterministic Python code
validates schemas, enforces outcome policy, applies only allowlisted edits, and
records provenance.
Each profile row contains:
Long trajectories are sent to the API through a turn-preserving evidence view.
Test observations, state-changing turns, malformed calls, final turns, patches,
and stable turn IDs receive priority. Shortened values include their original
character count and SHA-256 hash. This compaction affects only the API prompt;
classification never rewrites source JSONL or Parquet records.
- turn and role counts;
- canonical serialized characters and UTF-8 bytes;
- exact token count when a `tokenizer.json` is supplied;
- tool-call count and count by tool name;
- failed tool-call count and rate;
- observable error type and tool distributions;
- normalized error positions across five equal trajectory regions;
- early-error count and fraction;
- longest consecutive failed-tool-call run;
- malformed calls, unknown tools, missing results, and orphan results.
## API endpoint
The canonical token stream is compact, sorted JSON containing only `tools` and
`trajectory`. This is reproducible but is not presented as a universal chat
template. If training uses another serializer or tokenizer, rerun `profile`
with that exact tokenizer instead of converting characters to fake token counts.
The configured gateway is OpenAI chat-completions compatible:
## Heuristics
```text
POST https://llm-api.cowin.run/v1/chat/completions
```
The following conditions are hard rejections because they represent broken
training structure or an unambiguous repeated-failure pattern:
`/v1/text-completion` and `/v1/text-completions` resolve to the gateway's web
console rather than an inference API, so they are not used.
- malformed/unknown tool calls or broken assistant/tool pairing;
- at least five consecutive failed tool calls;
- at least five failed calls and a failure rate of at least 50%.
The default model is `glm-5.2`. The API key is read only from `GLM_API_KEY`. Do
not write a key into source code, command history, output manifests, or this
README.
Three weaker patterns are review flags, not automatic rejection:
## Project layout
- error count at or above the dataset-wide p99 threshold;
- at least eight failed calls, a failure rate of at least 20%, and errors in at
least four of five trajectory regions;
- at least three errors with at least 60% of all errors in the first 20% of tool
calls.
```text
swe_data_processing/
├── pyproject.toml
├── README.md
├── changelog.md # Versioned strategy changes
├── .env.example
├── src/swe_data_processing/
│ ├── cli.py # Command-line entry points
│ ├── client.py # GLM API client and JSON validation
│ ├── config.py # Environment-only runtime settings
│ ├── evidence.py # Prompt-only evidence compaction
│ ├── features.py # Deterministic static evidence extraction
│ ├── io.py # Streaming Parquet and JSONL readers
│ ├── policy.py # Immutable local policy guards
│ ├── repair.py # Allowlisted deterministic repair application
│ ├── resources.py # Packaged prompt/schema loading
│ ├── workflow.py # Classify, plan repair, and review stages
│ ├── prompts/ # Version-controlled GLM system prompts
│ └── schemas/ # JSON Schemas for every API stage
├── tests/ # Offline unit tests; no network calls
├── scripts/ # Existing sampling and profiling utilities
├── raw/Open-SWE-Traces/ # Downloaded dataset; ignored by Git
├── samples/ # Human-review samples
├── reports/ # Dataset reports and QC rubrics
└── qc_outputs/ # Generated manifests; ignored by Git
```
Length is separated from quality. Tokenized records are assigned to:
- `LE_81920`: fast 80K training subset;
- `81921_TO_131072`: 128K training subset;
- `131073_TO_262144`: deferred long-context subset;
- `GT_262144`: excluded from the default small-model training run.
The summary reports how much of the total token mass is contributed by the
longest 0.1%, 1%, and 5% of samples. This makes long-tail removal a measurable
dataset decision rather than a guess.
## Installation
The existing remote virtual environment can install the package in editable
mode:
```bash
cd /mnt/beegfs/yi/swe_data_processing
./.venv/bin/pip install -e '.[dev]'
cd /home/kxqandccx/kxq/tomlu/OpenSWETraces_cleanup
./.venv/bin/pip install -e '.[dev,tokens]'
```
For a fresh environment:
`tokenizers` is optional. Without it, profiling still produces turn, character,
byte, and tool-error metrics, but no token bucket decision.
## Full workflow
Use an explicit tokenizer already present on the machine:
```bash
python3 -m venv .venv
./.venv/bin/pip install --upgrade pip
./.venv/bin/pip install -e '.[dev]'
```
TOKENIZER_JSON=/path/to/tokenizer.json
mkdir -p qc_outputs/deterministic_v3
## Configuration
Export credentials in the shell that launches the pipeline:
```bash
export GLM_API_KEY='your-runtime-secret'
export GLM_API_BASE='https://llm-api.cowin.run'
export GLM_API_PATH='/v1/chat/completions'
export GLM_MODEL='glm-5.2'
```
Optional settings and their defaults:
```bash
export GLM_TIMEOUT_SECONDS=300
export GLM_MAX_RETRIES=5
export GLM_MAX_TOKENS=8192
export GLM_TEMPERATURE=0.0
export GLM_REASONING_EFFORT=high
export GLM_THINKING_ENABLED=true
```
`GLM_MAX_RETRIES=5` means one initial request plus at most five retries. The
client retries timeouts, connection failures, HTTP 429/5xx responses, malformed
JSON, and schema-invalid model output with bounded exponential backoff. HTTP
401/403 authentication failures are never retried.
If the gateway rejects GLM-specific `thinking` or `reasoning_effort` fields, the
client automatically retries with the portable OpenAI-compatible request subset.
## Commands
### 1. Verify API authentication and structured output
```bash
swe-qc smoke-test
```
The command prints the endpoint, model, request ID, usage, and a tiny validated
JSON response. It never prints the API key.
### 2. Extract deterministic evidence without API calls
From the 20-record review sample:
```bash
mkdir -p qc_outputs
swe-qc features \
--input samples/sample_20_seed_20260805.jsonl \
--output qc_outputs/sample20.features.jsonl \
--errors qc_outputs/sample20.features.errors.jsonl
```
From all Parquet shards:
```bash
swe-qc features \
swe-qc profile \
--input raw/Open-SWE-Traces \
--output qc_outputs/all.features.jsonl \
--errors qc_outputs/all.features.errors.jsonl \
--output qc_outputs/deterministic_v3/metrics.jsonl \
--errors qc_outputs/deterministic_v3/profile.errors.jsonl \
--tokenizer-json "$TOKENIZER_JSON" \
--workers 8 \
--resume
swe-qc summarize \
--metrics qc_outputs/deterministic_v3/metrics.jsonl \
--summary qc_outputs/deterministic_v3/summary.json \
--decisions qc_outputs/deterministic_v3/decisions.jsonl
swe-qc sample \
--input raw/Open-SWE-Traces \
--decisions qc_outputs/deterministic_v3/decisions.jsonl \
--output qc_outputs/deterministic_v3/review_sample.jsonl \
--per-group 20 \
--seed 20260818
```
Features include malformed tool arguments, unknown tools, role alternation,
state-changing turns, post-edit test evidence, masked shell pipelines, patch file
sets, patch size ratios, and explicit user constraints.
`profile --resume` skips sample IDs already present in the metrics file. Writes
are append-only and flushed per record. `summarize` and `sample` atomically
replace their outputs so a partial file is never mistaken for a complete run.
Any dataset subdirectory containing Parquet shards is also a valid `--input`,
so the four trajectory families can be profiled in separate CPU processes and
their metrics concatenated before `summarize`.
Profile rows and decisions record the absolute source Parquet shard. `sample`
uses that provenance to read only shards containing selected IDs instead of
scanning the complete dataset.
### 3. Classify trajectories through GLM-5.2
## Error detection boundary
Run a small pilot first:
```bash
swe-qc classify \
--input samples/sample_20_seed_20260805.jsonl \
--output qc_outputs/sample20.classifications.jsonl \
--errors qc_outputs/sample20.classification.errors.jsonl \
--limit 20 \
--resume
```
The classifier returns one of:
- `ACCEPT_SILVER_POSITIVE`
- `ACCEPT_NEGATIVE`
- `STATIC_REPAIR`
- `HOLD_UNVERIFIED`
- `REJECT`
Training use is tracked separately as `SFT_FULL`, `SFT_STEP_ONLY`,
`DPO_REJECTED`, `ERROR_ANALYSIS`, `HOLD`, or `DROP`.
### 4. Locate a safe prefix and score it independently
`audit` uses two isolated GLM calls for failed and unknown trajectories:
1. The boundary call receives the trajectory as roughly 32-turn blocks without
splitting an assistant action from its immediate tool result. It cannot see
`resolved` or reference patches, but it does see the trajectory's own final
model patch to check task coverage and diff pollution. It selects the earliest
unrepaired major/critical assistant turn, keeps a safe full process trace,
or returns `HOLD` when neither decision is supported.
2. Python slices the exact messages before that assistant turn. The quality call
receives only this materialized prefix, so suffix behavior cannot affect its
score.
For a complete trajectory, the quality call also sees its final model patch. A
truncated prefix never receives that suffix-derived patch. State-changing turns
are recorded as audit telemetry but are not automatic boundaries: writing code
is not itself an error, and recovered experiments are useful process data. The
effective boundary is the earliest unrecovered major or critical behavior found
by the boundary stage. The isolated quality stage rejects a retained prefix that
still contains an unrecovered severe problem.
Successful trajectories skip boundary selection and are scored as complete
trajectories. The command remains simple:
```bash
swe-qc audit \
--input samples/sample_20_seed_20260805.jsonl \
--output qc_outputs/sample20.audits.jsonl \
--errors qc_outputs/sample20.audit.errors.jsonl \
--workers 5 \
--resume
```
`--workers` bounds the number of records processed concurrently. JSONL writes
remain serialized in the main thread, so each completed record is appended
atomically even when API requests run in parallel. Output order follows request
completion order; `sample_id` remains the stable join key.
The gateway previously returned frequent HTTP 429 responses at 20 workers, so
five workers is the practical default for long runs unless the service limit is
raised.
The quality call lists concrete erroneous and inefficient assistant actions and
scores five 0-20 dimensions: planning, investigation, tool use and observation,
progress, and clarity/efficiency. Python recomputes issue counts, sums the five
dimensions, and assigns:
- `HIGH`: 80-100
- `MEDIUM`: 60-79
- `LOW`: 40-59
- `REJECT`: below 40, or an unrepaired major/critical problem remains
Incomplete prefixes can still be useful. Normal failed experiments are retained
when the agent later diagnoses, repairs, and verifies them. A prefix ending in a
complete tool result is valid process-SFT structure.
### 5. Create static repair plans
Only classifications that explicitly return `STATIC_REPAIR` are processed by
default:
```bash
swe-qc repair-plan \
--input samples/sample_20_seed_20260805.jsonl \
--classifications qc_outputs/sample20.classifications.jsonl \
--output qc_outputs/sample20.repair-plans.jsonl \
--errors qc_outputs/sample20.repair-plan.errors.jsonl \
--resume
```
The model may propose only operations from the allowlist. Code changes, test
result synthesis, patch replacement, and outcome upgrades are prohibited.
### 6. Apply plans deterministically
```bash
swe-qc apply-repair \
--input samples/sample_20_seed_20260805.jsonl \
--plans qc_outputs/sample20.repair-plans.jsonl \
--output qc_outputs/sample20.repaired.jsonl \
--errors qc_outputs/sample20.apply.errors.jsonl \
--resume
```
The applier checks target turns, read-only pair deletion, immutable patch fields,
and retained tool-output hashes. It emits a structured diff with input and output
SHA-256 hashes.
### 7. Independently review repairs
```bash
swe-qc review \
--input samples/sample_20_seed_20260805.jsonl \
--classifications qc_outputs/sample20.classifications.jsonl \
--plans qc_outputs/sample20.repair-plans.jsonl \
--repaired qc_outputs/sample20.repaired.jsonl \
--output qc_outputs/sample20.reviews.jsonl \
--errors qc_outputs/sample20.review.errors.jsonl \
--resume
```
Approval means that a static repair preserved evidence and structure. It does not
mean that the code patch was executed or proved correct.
## Output and resume behavior
Pipeline manifests are append-only JSONL. `--resume` reads completed sample IDs
from the output and skips them. Each API response includes non-sensitive
provenance:
- model and endpoint;
- gateway request ID;
- token usage when available;
- input SHA-256;
- compatibility-fallback flag;
- UTC creation time.
Errors are written as compact records containing sample ID, stage, exception
type, and a bounded message. Raw prompts and credentials are not copied into
error logs.
## Recommended rollout
1. Run offline tests.
2. Run `smoke-test` once.
3. Run classification on the 20 manually reviewed records.
4. Compare GLM decisions with the human labels.
5. Build a 2,000-record stratified calibration set.
6. Require at least 95% precision for `ACCEPT_SILVER_POSITIVE` before scaling.
7. Keep `resolved=0` as negative data rather than attempting to turn it into
positive trajectories.
8. Keep `resolved=-1` in an unverified manifest unless it contains explicit,
reliable failure evidence.
## Development
Run all offline tests:
```bash
./.venv/bin/pytest
```
Run lint checks:
```bash
./.venv/bin/ruff check src tests
```
Tests use an `httpx.MockTransport` and never contact the GLM endpoint.
## Credential handling
- `.env` files are ignored by Git.
- The provided API key is not stored anywhere in this project.
- Use a secret manager or a protected runtime environment variable for batch
jobs.
- Rotate the key if it has been copied into a public log, issue, or repository.
The detector intentionally favors precision over recall. It recognizes explicit
non-zero exit codes, test/build failure summaries, timeouts, permission errors,
missing commands/files, and tool exceptions. It does not treat the mere word
"error" as a failure, because tools often print source code or logs containing
that word. Human samples should be used to refine patterns only when the raw
tool observation provides an unambiguous signal.