56 lines
2.4 KiB
Markdown
56 lines
2.4 KiB
Markdown
# Changelog
|
|
|
|
This file records strategy changes that materially affect dataset decisions or
|
|
training-data semantics. Generated audit manifests are not treated as stable API
|
|
contracts.
|
|
|
|
## 2.0.0 - 2026-08-07
|
|
|
|
### Strategy
|
|
|
|
- Replaced the single-call trajectory audit with two isolated model calls:
|
|
boundary selection followed by prefix-only quality scoring.
|
|
- The boundary call receives an outcome-blind and patch-blind trajectory view.
|
|
Long trajectories are presented as ordered blocks targeting 32 turns, with
|
|
absolute turn IDs and immediate assistant/tool-result pairs kept together.
|
|
- Python materializes the exact prefix before the selected assistant turn. The
|
|
quality call is then built again from that prefix, so it cannot see the
|
|
removed suffix, external outcome, model patch, reference patch, or boundary
|
|
rationale.
|
|
- Simplified both prompts and added concrete positive, recovery, and late-claim
|
|
examples.
|
|
- Replaced the weighted six-dimension score and issue penalties with five
|
|
equally weighted 0-20 dimensions. Python computes the final 0-100 score and
|
|
tier.
|
|
- Distinguished a safe full process trajectory (`KEEP_FULL`) from an uncertain
|
|
trajectory (`HOLD`) so the boundary stage does not reject useful data merely
|
|
because no truncation is needed.
|
|
- Kept an explicit list of erroneous and inefficient assistant actions as a
|
|
quality surrogate; Python recomputes its counts locally.
|
|
|
|
### Validation
|
|
|
|
- A boundary must be an assistant turn inside the selected block.
|
|
- Boundary evidence must include an exact quote grounded in that assistant
|
|
turn.
|
|
- A prefix scorer cannot cite a turn outside the materialized prefix.
|
|
- `HIGH`, `MEDIUM`, `LOW`, and `REJECT` are computed locally rather than trusted
|
|
from model output.
|
|
|
|
### Motivation
|
|
|
|
- In the previous 500-record pilot, 360 of 423 prefix candidates included
|
|
suffix behavior in the score despite prompt instructions to score only the
|
|
retained prefix.
|
|
- Manual stress review found useful conservative cutoffs, but also late cutoffs
|
|
that retained failed patches and normal experiments incorrectly treated as
|
|
permanent errors.
|
|
- Prompt wording alone could not guarantee suffix isolation, so the new design
|
|
enforces it in the data passed to each call.
|
|
|
|
## 1.0.0 - 2026-08-06
|
|
|
|
- Initial static classification, one-call trajectory audit, allowlisted repair
|
|
planning, deterministic repair application, and independent repair review.
|
|
- Added API retries and concurrent JSONL processing.
|