Replace trajectory audit with isolated prefix scoring

2026-08-07 00:47:31 +08:00
parent f1a090e8c4
commit c6c583dcee
18 changed files with 914 additions and 469 deletions
@@ -0,0 +1,55 @@
+# Changelog
+
+This file records strategy changes that materially affect dataset decisions or
+training-data semantics. Generated audit manifests are not treated as stable API
+contracts.
+
+## 2.0.0 - 2026-08-07
+
+### Strategy
+
+- Replaced the single-call trajectory audit with two isolated model calls:
+  boundary selection followed by prefix-only quality scoring.
+- The boundary call receives an outcome-blind and patch-blind trajectory view.
+  Long trajectories are presented as ordered blocks targeting 32 turns, with
+  absolute turn IDs and immediate assistant/tool-result pairs kept together.
+- Python materializes the exact prefix before the selected assistant turn. The
+  quality call is then built again from that prefix, so it cannot see the
+  removed suffix, external outcome, model patch, reference patch, or boundary
+  rationale.
+- Simplified both prompts and added concrete positive, recovery, and late-claim
+  examples.
+- Replaced the weighted six-dimension score and issue penalties with five
+  equally weighted 0-20 dimensions. Python computes the final 0-100 score and
+  tier.
+- Distinguished a safe full process trajectory (`KEEP_FULL`) from an uncertain
+  trajectory (`HOLD`) so the boundary stage does not reject useful data merely
+  because no truncation is needed.
+- Kept an explicit list of erroneous and inefficient assistant actions as a
+  quality surrogate; Python recomputes its counts locally.
+
+### Validation
+
+- A boundary must be an assistant turn inside the selected block.
+- Boundary evidence must include an exact quote grounded in that assistant
+  turn.
+- A prefix scorer cannot cite a turn outside the materialized prefix.
+- `HIGH`, `MEDIUM`, `LOW`, and `REJECT` are computed locally rather than trusted
+  from model output.
+
+### Motivation
+
+- In the previous 500-record pilot, 360 of 423 prefix candidates included
+  suffix behavior in the score despite prompt instructions to score only the
+  retained prefix.
+- Manual stress review found useful conservative cutoffs, but also late cutoffs
+  that retained failed patches and normal experiments incorrectly treated as
+  permanent errors.
+- Prompt wording alone could not guarantee suffix isolation, so the new design
+  enforces it in the data passed to each call.
+
+## 1.0.0 - 2026-08-06
+
+- Initial static classification, one-call trajectory audit, allowlisted repair
+  planning, deterministic repair application, and independent repair review.
+- Added API retries and concurrent JSONL processing.