OpenSWETraces_cleanup/changelog.md

Changelog

This file records strategy changes that materially affect dataset decisions or training-data semantics. Generated audit manifests are not treated as stable API contracts.

Unreleased

Strategy correction

• Removed the unconditional cutoff before the first state-changing turn for failed and unverified trajectories. A write or edit is not evidence of an error by itself, and this cap discarded normal implementation, recovery, and verification work.
• The semantic boundary now remains the earliest unrecovered major or critical assistant behavior selected by the boundary stage. The isolated prefix-quality stage remains the safety gate. The first stateful turn is retained as telemetry for paired evaluation against the former policy.

2.0.0 - 2026-08-07

Strategy

• Replaced the single-call trajectory audit with two isolated model calls: boundary selection followed by prefix-only quality scoring.
• The boundary call receives an outcome-blind and reference-patch-blind view. It does receive the trajectory's own final model patch to check coverage and final diff pollution. Long trajectories are presented as ordered blocks targeting 32 turns, with absolute turn IDs and immediate assistant/tool-result pairs kept together.
• Python materializes the exact prefix before the selected assistant turn. The quality call is then built again from that prefix, so it cannot see the removed suffix, external outcome, model patch, reference patch, or boundary rationale.
• Simplified both prompts and added concrete positive, recovery, and late-claim examples.
• Replaced the weighted six-dimension score and issue penalties with five equally weighted 0-20 dimensions. Python computes the final 0-100 score and tier.
• Distinguished a safe full process trajectory (KEEP_FULL) from an uncertain trajectory (HOLD) so the boundary stage does not reject useful data merely because no truncation is needed.
• Kept an explicit list of erroneous and inefficient assistant actions as a quality surrogate; Python recomputes its counts locally.
• Each listed issue records whether it was recovered. Python rejects a model response that claims a prefix is safe while also listing an unrecovered major/critical issue.

Validation

• A boundary must be an assistant turn inside the selected block.
• Boundary evidence must include that assistant turn ID; Python attaches the exact immutable source excerpt rather than trusting model-copied quotes.
• The model does not repeat a block ID or prefix-safe boolean. Python derives the containing block from the validated boundary turn, while the isolated second call judges the materialized prefix itself.
• KEEP_FULL now requires explicit review of task coverage, exact interfaces, final repository state, user scope constraints, and claim/tool consistency; passing tests alone are not treated as proof.
• Complete-trajectory scoring receives the final model patch, while truncated prefix scoring remains suffix-isolated.
• Failed and unverified (resolved=0/-1) records are capped before the first statically detected state-changing assistant turn. The earlier of that cap and the GLM boundary is materialized, preferring shorter safe investigation traces over late cuts.
• Removed redundant model-written prefix-validity booleans. Python derives safety directly from unrecovered major/critical behavior issues.
• A prefix scorer cannot cite a turn outside the materialized prefix.
• HIGH, MEDIUM, LOW, and REJECT are computed locally rather than trusted from model output.

Stress pilot

• A difficulty-weighted 15-record set was used to expose late-boundary errors; it is not an estimate of the full dataset acceptance rate.
• Raw semantic boundaries were often too late or incorrectly returned KEEP_FULL. After applying the non-success stateful cap, two independent human reviewers found all 15 effective prefixes safe and no late boundary.
• Retained-turn ratios ranged from 11.14% to 39.74%, with an 18.73% median. The final quality distribution was 10 HIGH and 5 MEDIUM.
• The final low-reasoning, five-worker run completed 15/15 records without an error in 90.05 seconds. High-reasoning mode was substantially slower and had gateway timeouts, so it is not the practical batch default.

Motivation

• In the previous 500-record pilot, 360 of 423 prefix candidates included suffix behavior in the score despite prompt instructions to score only the retained prefix.
• Manual stress review found useful conservative cutoffs, but also late cutoffs that retained failed patches and normal experiments incorrectly treated as permanent errors.
• Prompt wording alone could not guarantee suffix isolation, so the new design enforces it in the data passed to each call.

1.0.0 - 2026-08-06

• Initial static classification, one-call trajectory audit, allowlisted repair planning, deterministic repair application, and independent repair review.
• Added API retries and concurrent JSONL processing.