Harden non-success trajectory truncation

README.md

@@ -210,13 +210,21 @@ Training use is tracked separately as `SFT_FULL`, `SFT_STEP_ONLY`,
 
 1. The boundary call receives the trajectory as roughly 32-turn blocks without
    splitting an assistant action from its immediate tool result. It cannot see
-   `resolved`, model patches, or reference patches. It selects the earliest
+   `resolved` or reference patches, but it does see the trajectory's own final
+   model patch to check task coverage and diff pollution. It selects the earliest
    unrepaired major/critical assistant turn, keeps a safe full process trace,
    or returns `HOLD` when neither decision is supported.
 2. Python slices the exact messages before that assistant turn. The quality call
    receives only this materialized prefix, so suffix behavior cannot affect its
    score.
 
+For a complete trajectory, the quality call also sees its final model patch. A
+truncated prefix never receives that suffix-derived patch. Every non-success
+trajectory (`resolved=0` or `-1`) is capped before its first statically detected
+state-changing assistant turn. The effective boundary is the earlier of this
+local cap and the GLM boundary. This intentionally prefers a shorter, safer
+investigation trace.
+
 Successful trajectories skip boundary selection and are scored as complete
 trajectories. The command remains simple:
 

changelog.md

@@ -10,7 +10,9 @@ contracts.
 
 - Replaced the single-call trajectory audit with two isolated model calls:
   boundary selection followed by prefix-only quality scoring.
-- The boundary call receives an outcome-blind and patch-blind trajectory view.
+- The boundary call receives an outcome-blind and reference-patch-blind view.
+  It does receive the trajectory's own final model patch to check coverage and
+  final diff pollution.
   Long trajectories are presented as ordered blocks targeting 32 turns, with
   absolute turn IDs and immediate assistant/tool-result pairs kept together.
 - Python materializes the exact prefix before the selected assistant turn. The
@@ -27,16 +29,46 @@ contracts.
   because no truncation is needed.
 - Kept an explicit list of erroneous and inefficient assistant actions as a
   quality surrogate; Python recomputes its counts locally.
+- Each listed issue records whether it was recovered. Python rejects a model
+  response that claims a prefix is safe while also listing an unrecovered
+  major/critical issue.
 
 ### Validation
 
 - A boundary must be an assistant turn inside the selected block.
-- Boundary evidence must include an exact quote grounded in that assistant
+- Boundary evidence must include that assistant turn ID; Python attaches the
-  turn.
+  exact immutable source excerpt rather than trusting model-copied quotes.
+- The model does not repeat a block ID or prefix-safe boolean. Python derives
+  the containing block from the validated boundary turn, while the isolated
+  second call judges the materialized prefix itself.
+- `KEEP_FULL` now requires explicit review of task coverage, exact interfaces,
+  final repository state, user scope constraints, and claim/tool consistency;
+  passing tests alone are not treated as proof.
+- Complete-trajectory scoring receives the final model patch, while truncated
+  prefix scoring remains suffix-isolated.
+- Failed and unverified (`resolved=0/-1`) records are capped before the first
+  statically detected state-changing assistant turn. The earlier of that cap
+  and the GLM boundary is materialized, preferring shorter safe investigation
+  traces over late cuts.
+- Removed redundant model-written prefix-validity booleans. Python derives
+  safety directly from unrecovered major/critical behavior issues.
 - A prefix scorer cannot cite a turn outside the materialized prefix.
 - `HIGH`, `MEDIUM`, `LOW`, and `REJECT` are computed locally rather than trusted
   from model output.
 
+### Stress pilot
+
+- A difficulty-weighted 15-record set was used to expose late-boundary errors;
+  it is not an estimate of the full dataset acceptance rate.
+- Raw semantic boundaries were often too late or incorrectly returned
+  `KEEP_FULL`. After applying the non-success stateful cap, two independent
+  human reviewers found all 15 effective prefixes safe and no late boundary.
+- Retained-turn ratios ranged from 11.14% to 39.74%, with an 18.73% median.
+  The final quality distribution was 10 `HIGH` and 5 `MEDIUM`.
+- The final low-reasoning, five-worker run completed 15/15 records without an
+  error in 90.05 seconds. High-reasoning mode was substantially slower and had
+  gateway timeouts, so it is not the practical batch default.
+
 ### Motivation
 
 - In the previous 500-record pilot, 360 of 423 prefix candidates included

src/swe_data_processing/audit.py

@@ -2,11 +2,9 @@
 
 from __future__ import annotations
 
-import json
-import re
 from typing import Any
 
-from .features import index_trajectory
+from .features import extract_static_signals, index_trajectory
 from .io import get_sample_id
 from .policy import PolicyViolation
 
@@ -19,30 +17,8 @@ QUALITY_DIMENSIONS = (
 )
 
 
-def _flatten_text(value: Any) -> str:
-    if isinstance(value, str):
-        return value
-    if isinstance(value, dict):
-        return " ".join(_flatten_text(item) for item in value.values())
-    if isinstance(value, list):
-        return " ".join(_flatten_text(item) for item in value)
-    return json.dumps(value, ensure_ascii=False, default=str)
-
-
-def _normalized_text(value: Any) -> str:
-    return re.sub(r"\s+", " ", _flatten_text(value)).strip()
-
-
-def _evidence_is_grounded(message: dict[str, Any], quote: str) -> bool:
-    """Return whether a short evidence quote occurs in the referenced message."""
-
-    message_text = _normalized_text({key: value for key, value in message.items() if key != "turn_id"})
-    return _normalized_text(quote) in message_text
-
-
 def validate_boundary(
     record: dict[str, Any],
-    blocks: list[dict[str, Any]],
     result: dict[str, Any],
 ) -> None:
     """Reject structurally inconsistent or ungrounded boundary decisions."""
@@ -52,45 +28,36 @@ def validate_boundary(
     assistant_turns = {
         message["turn_id"] for message in trajectory if message.get("role") == "assistant"
     }
-    block_ranges = {
-        block["block_id"]: range(block["start_turn"], block["end_turn"] + 1)
-        for block in blocks
-    }
     if result["sample_id"] != get_sample_id(record):
         raise PolicyViolation("Boundary sample_id does not match the source record")
 
     decision = result["decision"]
-    block_id = result["candidate_block_id"]
     boundary = result["truncate_before_turn"]
     if decision in {"KEEP_FULL", "HOLD"}:
-        if block_id is not None or boundary is not None:
+        if boundary is not None:
             raise PolicyViolation(f"A {decision} decision cannot contain a boundary")
         if result["category"] != "NONE" or result["severity"] != "NONE":
             raise PolicyViolation(f"A {decision} decision must use NONE category and severity")
-        if decision == "KEEP_FULL" and not result["prefix_safe_before_boundary"]:
+        if decision == "KEEP_FULL" and result["checks"] != {
-            raise PolicyViolation("KEEP_FULL requires a safe full trajectory")
+            "task_coverage": "COMPLETE",
+            "final_patch_scope": "CLEAN",
+            "constraints": "RESPECTED",
+            "claims_vs_observations": "CONSISTENT",
+        }:
+            raise PolicyViolation("KEEP_FULL requires all four checks to pass")
         return
 
-    if block_id not in block_ranges:
-        raise PolicyViolation("candidate_block_id does not reference an input block")
     if boundary not in assistant_turns:
         raise PolicyViolation("truncate_before_turn must reference an assistant turn")
-    if boundary not in block_ranges[block_id]:
-        raise PolicyViolation("truncate_before_turn must be inside candidate_block_id")
-    if not result["prefix_safe_before_boundary"]:
-        raise PolicyViolation("A truncation boundary requires a safe retained prefix")
     if result["severity"] not in {"MAJOR", "CRITICAL"}:
         raise PolicyViolation("A truncation boundary requires MAJOR or CRITICAL severity")
 
-    evidence = result["evidence"]
+    evidence_turns = result["evidence_turns"]
-    if not any(item["turn_id"] == boundary for item in evidence):
+    if boundary not in evidence_turns:
-        raise PolicyViolation("Boundary evidence must quote the excluded assistant turn")
+        raise PolicyViolation("Boundary evidence must include the excluded assistant turn")
-    for item in evidence:
+    for turn_id in evidence_turns:
-        turn_id = item["turn_id"]
         if turn_id not in messages:
             raise PolicyViolation("Boundary evidence references a missing turn")
-        if not _evidence_is_grounded(messages[turn_id], item["quote"]):
-            raise PolicyViolation("Boundary evidence quote is not present in its source turn")
 
 
 def materialize_prefix(
@@ -102,6 +69,45 @@ def materialize_prefix(
     return list(trajectory if boundary is None else trajectory[: boundary - 1])
 
 
+def effective_boundary_policy(
+    record: dict[str, Any], model_result: dict[str, Any]
+) -> dict[str, Any]:
+    """Choose a conservative effective boundary for one trajectory.
+
+    A failed or unverified record may not retain a statically detected
+    state-changing assistant turn. This deliberately favors a shorter
+    investigation prefix over a late semantic boundary that could retain a bad
+    implementation.
+    """
+
+    model_boundary = model_result["truncate_before_turn"]
+    first_stateful = min(
+        extract_static_signals(record)["stateful_turns"], default=None
+    )
+    candidates = []
+    if model_result["decision"] == "TRUNCATE":
+        candidates.append((model_boundary, "MODEL_BOUNDARY"))
+    if int(record.get("resolved", -1)) != 1 and first_stateful is not None:
+        candidates.append((first_stateful, "NON_SUCCESS_FIRST_STATEFUL_CAP"))
+
+    if candidates:
+        boundary, source = min(candidates, key=lambda item: item[0])
+        return {
+            "decision": "TRUNCATE",
+            "truncate_before_turn": boundary,
+            "source": source,
+            "model_truncate_before_turn": model_boundary,
+            "first_stateful_turn": first_stateful,
+        }
+    return {
+        "decision": model_result["decision"],
+        "truncate_before_turn": None,
+        "source": "MODEL_DECISION",
+        "model_truncate_before_turn": model_boundary,
+        "first_stateful_turn": first_stateful,
+    }
+
+
 def validate_prefix_quality(
     sample_id: str,
     prefix: list[dict[str, Any]],
@@ -111,31 +117,67 @@ def validate_prefix_quality(
 
     if result["sample_id"] != sample_id:
         raise PolicyViolation("Quality sample_id does not match the source record")
-    if result["prefix_valid"] == result["unrecovered_major_or_critical"]:
-        raise PolicyViolation("prefix_valid contradicts unrecovered_major_or_critical")
-    if result["unrecovered_major_or_critical"] and not result["evidence"]:
-        raise PolicyViolation("An unsafe prefix requires grounded evidence")
 
     messages = {message["turn_id"]: message for message in index_trajectory({"trajectory": prefix})}
-    assistant_turns = {
-        turn_id for turn_id, message in messages.items() if message.get("role") == "assistant"
-    }
     for issue in result["behavior_issues"]:
-        if issue["assistant_turn"] not in assistant_turns:
+        if issue["turn_id"] not in messages:
-            raise PolicyViolation("Behavior issue references an assistant turn outside the prefix")
+            raise PolicyViolation("Behavior issue references a turn outside the prefix")
-    for item in result["evidence"]:
+    unrecovered_severe = [
-        turn_id = item["turn_id"]
+        issue
+        for issue in result["behavior_issues"]
+        if issue["severity"] in {"MAJOR", "CRITICAL"} and not issue["recovered"]
+    ]
+    if unrecovered_severe and not any(
+        issue["turn_id"] in result["evidence_turns"] for issue in unrecovered_severe
+    ):
+        raise PolicyViolation("Unsafe prefix evidence must include an unrecovered severe issue")
+    for turn_id in result["evidence_turns"]:
         if turn_id not in messages:
             raise PolicyViolation("Quality evidence references a turn outside the prefix")
-        if not _evidence_is_grounded(messages[turn_id], item["quote"]):
+
-            raise PolicyViolation("Quality evidence quote is not present in its source turn")
+
+def derive_prefix_safety(result: dict[str, Any]) -> dict[str, Any]:
+    """Derive safety from model-listed issues instead of redundant booleans."""
+
+    unrecovered_severe = [
+        issue
+        for issue in result["behavior_issues"]
+        if issue["severity"] in {"MAJOR", "CRITICAL"} and not issue["recovered"]
+    ]
+    return {
+        "prefix_valid": not unrecovered_severe,
+        "unrecovered_major_or_critical": bool(unrecovered_severe),
+    }
+
+
+def materialize_evidence(
+    trajectory: list[dict[str, Any]], turn_ids: list[int]
+) -> list[dict[str, Any]]:
+    """Attach deterministic source excerpts to model-selected turn IDs."""
+
+    messages = {
+        message["turn_id"]: message
+        for message in index_trajectory({"trajectory": trajectory})
+    }
+    evidence = []
+    for turn_id in turn_ids:
+        message = messages[turn_id]
+        value = message.get("content")
+        if not value and message.get("tool_calls"):
+            value = message["tool_calls"]
+        excerpt = str(value)[:600]
+        evidence.append(
+            {"turn_id": turn_id, "role": message.get("role"), "excerpt": excerpt}
+        )
+    return evidence
 
 
 def compute_prefix_quality(result: dict[str, Any]) -> dict[str, Any]:
     """Compute the final score and tier from five model-provided dimensions."""
 
     score = sum(int(result["dimensions"][name]) for name in QUALITY_DIMENSIONS)
-    if not result["prefix_valid"] or result["unrecovered_major_or_critical"] or score < 40:
+    safety = derive_prefix_safety(result)
+    if not safety["prefix_valid"] or score < 40:
         tier = "REJECT"
     elif score >= 80:
         tier = "HIGH"
@@ -153,6 +195,8 @@ def compute_prefix_quality(result: dict[str, Any]) -> dict[str, Any]:
             "minor": sum(issue["severity"] == "MINOR" for issue in issues),
             "major": sum(issue["severity"] == "MAJOR" for issue in issues),
             "critical": sum(issue["severity"] == "CRITICAL" for issue in issues),
+            "recovered": sum(issue["recovered"] for issue in issues),
+            "unrecovered": sum(not issue["recovered"] for issue in issues),
         },
         "formula_version": "five-equal-dimensions-v2",
     }

src/swe_data_processing/prompts/boundary.md

@@ -3,6 +3,23 @@ You are finding a safe truncation point in a coding-agent trajectory. You cannot
 The trajectory is split into ordered blocks. Turn IDs are absolute. Find the earliest assistant turn B that begins
 a MAJOR or CRITICAL problem which is never repaired. The saved training prefix will contain only turns before B.
 
+Before deciding, check the whole trajectory against the actual user request:
+
+- Did the final implementation satisfy every requested behavior and exact interface, not just a small subtask?
+- Does the last visible diff/status leave tests, generated files, dependency files, summaries, or unrelated edits?
+- Did the agent violate a literal scope constraint such as making only non-test changes?
+- Do the last build/test/tool observations contradict the final claims or show an unrepaired broken repository?
+
+`final_model_patch` is the actual final diff left by this trajectory, not a reference answer. Use it to verify task
+coverage and final scope. Test files, dependency files, generated files, and unrelated changes in that diff are real
+final state even when the assistant says they were cleaned up.
+
+Passing existing tests is useful evidence, but it does not prove that a missing requirement or wrong public API is
+correct. Re-read the request before choosing KEEP_FULL.
+
+Fill all four checks before the decision. KEEP_FULL is allowed only when they are COMPLETE, CLEAN, RESPECTED, and
+CONSISTENT. If a check fails, locate the earliest causal assistant turn for TRUNCATE; use HOLD when no safe B is clear.
+
 Choose TRUNCATE only when both are true:
 
 1. Turn B clearly starts a serious problem, such as an unrepaired harmful edit, a persistent wrong implementation,
@@ -13,6 +30,9 @@ Return KEEP_FULL when the whole trajectory has no unrepaired MAJOR or CRITICAL p
 failed experiments or minor inefficiency. Return HOLD only when the evidence is incomplete or contradictory enough
 that you cannot establish either a safe full trajectory or a safe prefix.
 
+Never choose a command that merely reveals a problem as B. Choose the earlier assistant edit or decision that caused
+the bad state. If you can see a major final defect but cannot identify a safe causal assistant turn, return HOLD.
+
 Keep normal debugging. A failed command, a plausible experiment, or temporarily broken code is not a boundary when
 the agent later diagnoses, repairs, and verifies it. Do not punish harmless inefficiency.
 
@@ -22,8 +42,8 @@ assistant turn that created the bad patch. Do not merely remove the final answer
 Read user constraints literally. Do not invent stronger restrictions. Reading tests, running tests, or creating a
 requested reproduction script is not the same as modifying repository tests.
 
-Evidence must include a short exact quote from assistant turn B. Supporting quotes may come from later tool results.
+Evidence turn IDs must include assistant turn B. Supporting turn IDs may point to later tool results. Python will attach
-The candidate block must be the block that contains B.
+exact source excerpts and derive the containing block, so do not reproduce quotes or block IDs.
 
 Positive example: turn 40 writes code that calls a nonexistent method, turn 46 shows the resulting failure, and the
 edit is never repaired. Choose turn 40.

src/swe_data_processing/prompts/prefix_quality.md

@@ -1,13 +1,30 @@
 You are scoring process-SFT data. You can see only the exact prefix that would be kept for training. You cannot run
 code, and you know nothing about any removed suffix or external outcome.
 
-First decide whether this prefix itself contains an unrepaired MAJOR or CRITICAL harmful behavior. If it does, mark
+First identify every concrete error or inefficiency and say whether it was recovered. Cite an unrecovered MAJOR or
-the prefix invalid and quote the harmful turn. Otherwise mark it valid.
+CRITICAL issue in `evidence_turns`. Python derives prefix validity from this issue list and attaches exact excerpts.
 
-Also list concrete erroneous or inefficient assistant actions. An ERROR is a wrong tool call, edit, interpretation,
+Check four things before scoring:
-or claim. An INEFFICIENCY is avoidable repetition, noise, or a clearly wasteful detour. A reasonable experiment that
+
-fails and is then read and handled correctly is not automatically an error. Recovered mistakes may be listed, but
+- coverage: visible work follows the actual requested behavior and exact interfaces;
-they do not by themselves make the prefix invalid.
+- current state: edits do not leave known compile errors, failing callers, or repository pollution;
+- constraints: literal scope rules such as changing only non-test files are respected;
+- honesty: conclusions match the latest relevant tool observations.
+
+A passing test does not excuse a missing requirement, wrong API, unrelated diff, or violated scope constraint.
+When `final_model_patch` is present, it is the exact final diff for this full trajectory and should be checked. It is
+omitted for a truncated prefix because later edits are intentionally hidden.
+
+If the user says test changes were already handled, an old visible test that expects the old behavior may be stale.
+Do not call a correct source change unsafe only for that mismatch. Still treat modifying those test files as a scope
+violation when the user asked for minimal non-test changes.
+
+Also list concrete erroneous or inefficient behavior. Anchor each issue to the most useful visible turn: usually the
+assistant action, or its tool-result turn when the observed failure is the clearest evidence. An ERROR is a wrong tool
+call, edit, interpretation, or claim. An INEFFICIENCY is avoidable repetition, noise, or a clearly wasteful detour. A
+reasonable experiment that fails and is then read and handled correctly is not automatically an error. For every
+issue, say whether the bad state was later recovered. Recovered mistakes may be listed, but they do not by themselves
+make the prefix invalid. Any unrecovered MAJOR or CRITICAL issue makes the prefix invalid.
 
 An incomplete prefix can still be useful. It may teach problem understanding, repository exploration, reproduction,
 tool use, or correct reading of tool results. It does not need to solve the issue. Ending after a tool result is valid.
@@ -28,3 +45,6 @@ ends before implementation. It can be MEDIUM or HIGH even though it is incomplet
 
 Negative example: the prefix itself contains an unreverted wrong-file edit and then ends. Mark it invalid even if
 the earlier investigation was useful.
+
+Broken-state example: the prefix changes a function signature, has not updated known callers, and ends after a build
+failure. It is invalid. Incomplete investigation with no harmful edit can still be valid.

src/swe_data_processing/schemas/boundary_output.schema.json

@@ -3,16 +3,26 @@
   "type": "object",
   "additionalProperties": false,
   "required": [
-    "sample_id", "decision", "candidate_block_id", "truncate_before_turn",
+    "sample_id", "checks", "decision", "truncate_before_turn", "category",
-    "prefix_safe_before_boundary", "category", "severity", "state_effect",
+    "severity", "state_effect", "evidence_turns", "reason"
-    "evidence", "reason"
   ],
   "properties": {
     "sample_id": {"type": "string", "minLength": 1},
+    "checks": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": [
+        "task_coverage", "final_patch_scope", "constraints", "claims_vs_observations"
+      ],
+      "properties": {
+        "task_coverage": {"enum": ["COMPLETE", "INCOMPLETE", "UNCLEAR"]},
+        "final_patch_scope": {"enum": ["CLEAN", "POLLUTED", "UNCLEAR"]},
+        "constraints": {"enum": ["RESPECTED", "VIOLATED", "UNCLEAR"]},
+        "claims_vs_observations": {"enum": ["CONSISTENT", "CONTRADICTED", "UNCLEAR"]}
+      }
+    },
     "decision": {"enum": ["TRUNCATE", "KEEP_FULL", "HOLD"]},
-    "candidate_block_id": {"type": ["string", "null"], "pattern": "^block-[0-9]{3}$"},
     "truncate_before_turn": {"type": ["integer", "null"], "minimum": 1},
-    "prefix_safe_before_boundary": {"type": "boolean"},
     "category": {
       "enum": [
         "NONE", "USER_CONSTRAINT_VIOLATION", "HARMFUL_STATE_CHANGE",
@@ -22,22 +32,13 @@
     },
     "severity": {"enum": ["NONE", "MAJOR", "CRITICAL"]},
     "state_effect": {"enum": ["NONE", "UNRECOVERED", "UNCLEAR"]},
-    "evidence": {
+    "evidence_turns": {
       "type": "array",
+      "items": {"type": "integer", "minimum": 1},
+      "uniqueItems": true,
       "maxItems": 4,
-      "items": {"$ref": "#/$defs/evidence"}
+      "default": []
     },
     "reason": {"type": "string", "minLength": 1, "maxLength": 1000}
-  },
-  "$defs": {
-    "evidence": {
-      "type": "object",
-      "additionalProperties": false,
-      "required": ["turn_id", "quote"],
-      "properties": {
-        "turn_id": {"type": "integer", "minimum": 1},
-        "quote": {"type": "string", "minLength": 1, "maxLength": 600}
-      }
-    }
   }
 }

src/swe_data_processing/schemas/prefix_quality_output.schema.json

@@ -3,13 +3,10 @@
   "type": "object",
   "additionalProperties": false,
   "required": [
-    "sample_id", "prefix_valid", "unrecovered_major_or_critical",
+    "sample_id", "behavior_issues", "dimensions", "evidence_turns", "reason"
-    "behavior_issues", "dimensions", "evidence", "reason"
   ],
   "properties": {
     "sample_id": {"type": "string", "minLength": 1},
-    "prefix_valid": {"type": "boolean"},
-    "unrecovered_major_or_critical": {"type": "boolean"},
     "behavior_issues": {
       "type": "array",
       "maxItems": 20,
@@ -30,10 +27,12 @@
         "clarity_and_efficiency": {"type": "integer", "minimum": 0, "maximum": 20}
       }
     },
-    "evidence": {
+    "evidence_turns": {
       "type": "array",
+      "items": {"type": "integer", "minimum": 1},
+      "uniqueItems": true,
       "maxItems": 4,
-      "items": {"$ref": "#/$defs/evidence"}
+      "default": []
     },
     "reason": {"type": "string", "minLength": 1, "maxLength": 800}
   },
@@ -41,21 +40,13 @@
     "behavior_issue": {
       "type": "object",
       "additionalProperties": false,
-      "required": ["assistant_turn", "kind", "severity", "reason"],
+      "required": ["turn_id", "kind", "severity", "recovered", "reason"],
-      "properties": {
-        "assistant_turn": {"type": "integer", "minimum": 1},
-        "kind": {"enum": ["ERROR", "INEFFICIENCY"]},
-        "severity": {"enum": ["MINOR", "MAJOR", "CRITICAL"]},
-        "reason": {"type": "string", "minLength": 1, "maxLength": 500}
-      }
-    },
-    "evidence": {
-      "type": "object",
-      "additionalProperties": false,
-      "required": ["turn_id", "quote"],
       "properties": {
         "turn_id": {"type": "integer", "minimum": 1},
-        "quote": {"type": "string", "minLength": 1, "maxLength": 600}
+        "kind": {"enum": ["ERROR", "INEFFICIENCY"]},
+        "severity": {"enum": ["MINOR", "MAJOR", "CRITICAL"]},
+        "recovered": {"type": "boolean"},
+        "reason": {"type": "string", "minLength": 1, "maxLength": 500}
       }
     }
   }

src/swe_data_processing/workflow.py

@@ -9,6 +9,9 @@ from typing import Any
 
 from .audit import (
     compute_prefix_quality,
+    derive_prefix_safety,
+    effective_boundary_policy,
+    materialize_evidence,
     materialize_prefix,
     validate_boundary,
     validate_prefix_quality,
@@ -121,7 +124,7 @@ def _trajectory_only_record(
 
 
 def prepare_boundary_payload(record: dict[str, Any]) -> dict[str, Any]:
-    """Build an outcome-blind, patch-blind block view for boundary selection."""
+    """Build an outcome-blind, reference-patch-blind boundary payload."""
 
     raw_trajectory = record.get("trajectory") or []
     indexed = index_trajectory(record)
@@ -132,6 +135,9 @@ def prepare_boundary_payload(record: dict[str, Any]) -> dict[str, Any]:
         "user_request": compact_text(_first_user_issue(indexed), 16_000),
         "allowed_tool_names": signals["allowed_tool_names"],
         "trajectory_blocks": blocks,
+        "final_model_patch": compact_patch_object(
+            (record.get("metadata") or {}).get("model_patch") or {}
+        ),
         "prompt_compaction": compaction,
     }
 
@@ -146,7 +152,7 @@ def prepare_prefix_quality_payload(
     signals = extract_static_signals(prefix_record)
     trajectory, compaction = compact_trajectory(indexed, signals)
     assistant_turns = sum(message.get("role") == "assistant" for message in prefix)
-    return {
+    payload = {
         "sample_id": get_sample_id(record),
         "user_request": compact_text(_first_user_issue(indexed), 16_000),
         "allowed_tool_names": signals["allowed_tool_names"],
@@ -159,6 +165,11 @@ def prepare_prefix_quality_payload(
         },
         "prompt_compaction": compaction,
     }
+    if len(prefix) == len(record.get("trajectory") or []):
+        payload["final_model_patch"] = compact_patch_object(
+            (record.get("metadata") or {}).get("model_patch") or {}
+        )
+    return payload
 
 
 def audit_trajectory(record: dict[str, Any], client: GLMClient) -> dict[str, Any]:
@@ -173,6 +184,7 @@ def audit_trajectory(record: dict[str, Any], client: GLMClient) -> dict[str, Any
     if resolved == 1:
         prefix = materialize_prefix(record, None)
         evaluation_mode = "FULL_TRAJECTORY"
+        effective_boundary = None
     else:
         boundary_payload = prepare_boundary_payload(record)
         boundary_response = client.invoke_json(
@@ -180,26 +192,39 @@ def audit_trajectory(record: dict[str, Any], client: GLMClient) -> dict[str, Any
             payload=boundary_payload,
             schema=load_schema("boundary_output.schema.json"),
         )
-        blocks = boundary_payload["trajectory_blocks"]
+        validate_boundary(record, boundary_response.data)
-        validate_boundary(record, blocks, boundary_response.data)
         boundary_result = dict(boundary_response.data)
+        boundary = boundary_result["truncate_before_turn"]
+        boundary_result["candidate_block_id"] = next(
+            (
+                block["block_id"]
+                for block in boundary_payload["trajectory_blocks"]
+                if boundary is not None
+                and block["start_turn"] <= boundary <= block["end_turn"]
+            ),
+            None,
+        )
+        boundary_result["grounded_evidence"] = materialize_evidence(
+            source_trajectory, boundary_result["evidence_turns"]
+        )
+        effective_boundary = effective_boundary_policy(record, boundary_result)
         boundary_provenance = _provenance(
             "boundary", boundary_response, boundary_payload, client
         )
         evaluation_mode = "PROCESS_PREFIX"
-        if boundary_result["decision"] == "HOLD":
+        if effective_boundary["decision"] == "HOLD":
             return {
                 "sample_id": sample_id,
                 "pipeline_version": "2.0",
                 "evaluation_mode": evaluation_mode,
-                "boundary": boundary_result,
+                "boundary": {"model": boundary_result, "effective": effective_boundary},
                 "prefix": None,
                 "quality": None,
                 "recommended_use": "HOLD",
                 "source_record": _audit_source_record(record),
                 "provenance": {"boundary": boundary_provenance, "quality": None},
             }
-        prefix = materialize_prefix(record, boundary_result["truncate_before_turn"])
+        prefix = materialize_prefix(record, effective_boundary["truncate_before_turn"])
 
     quality_payload = prepare_prefix_quality_payload(record, prefix)
     quality_response = client.invoke_json(
@@ -209,23 +234,31 @@ def audit_trajectory(record: dict[str, Any], client: GLMClient) -> dict[str, Any
     )
     validate_prefix_quality(sample_id, prefix, quality_response.data)
     quality = dict(quality_response.data)
+    quality["grounded_evidence"] = materialize_evidence(
+        prefix, quality["evidence_turns"]
+    )
+    quality["local_safety"] = derive_prefix_safety(quality)
     quality["local_score"] = compute_prefix_quality(quality)
     tier = quality["local_score"]["quality_tier"]
     if tier == "REJECT":
         recommended_use = "REJECT"
     elif resolved == 1:
         recommended_use = "FULL_TRAJECTORY_CANDIDATE"
+    elif resolved == 0 and effective_boundary["decision"] == "KEEP_FULL":
+        recommended_use = "HOLD"
     else:
         recommended_use = "PROCESS_PREFIX_CANDIDATE"
 
-    boundary_turn = (
+    boundary_turn = effective_boundary["truncate_before_turn"] if effective_boundary else None
-        boundary_result["truncate_before_turn"] if boundary_result is not None else None
-    )
     return {
         "sample_id": sample_id,
         "pipeline_version": "2.0",
         "evaluation_mode": evaluation_mode,
-        "boundary": boundary_result,
+        "boundary": (
+            {"model": boundary_result, "effective": effective_boundary}
+            if boundary_result is not None
+            else None
+        ),
         "prefix": {
             "truncate_before_turn": boundary_turn,
             "source_turn_count": len(source_trajectory),

tests/test_audit.py

@@ -8,6 +8,8 @@ import pytest
 
 from swe_data_processing.audit import (
     compute_prefix_quality,
+    derive_prefix_safety,
+    effective_boundary_policy,
     materialize_prefix,
     validate_boundary,
     validate_prefix_quality,
@@ -38,27 +40,25 @@ def _record() -> dict:
 def _boundary() -> dict:
     return {
         "sample_id": "sample-1",
+        "checks": {
+            "task_coverage": "COMPLETE",
+            "final_patch_scope": "CLEAN",
+            "constraints": "RESPECTED",
+            "claims_vs_observations": "CONSISTENT",
+        },
         "decision": "TRUNCATE",
-        "candidate_block_id": "block-001",
         "truncate_before_turn": 5,
-        "prefix_safe_before_boundary": True,
         "category": "PERSISTENT_WRONG_IMPLEMENTATION",
         "severity": "MAJOR",
         "state_effect": "UNRECOVERED",
-        "evidence": [{"turn_id": 5, "quote": "apply the harmful patch"}],
+        "evidence_turns": [5],
         "reason": "The patch fails and is not repaired.",
     }
 
 
-def _blocks() -> list[dict]:
-    return [{"block_id": "block-001", "start_turn": 1, "end_turn": 6}]
-
-
 def _quality() -> dict:
     return {
         "sample_id": "sample-1",
-        "prefix_valid": True,
-        "unrecovered_major_or_critical": False,
         "behavior_issues": [],
         "dimensions": {
             "planning": 14,
@@ -67,13 +67,13 @@ def _quality() -> dict:
             "progress": 12,
             "clarity_and_efficiency": 10,
         },
-        "evidence": [],
+        "evidence_turns": [],
         "reason": "Useful investigation.",
     }
 
 
 def test_valid_boundary_materializes_exact_prefix() -> None:
-    validate_boundary(_record(), _blocks(), _boundary())
+    validate_boundary(_record(), _boundary())
     prefix = materialize_prefix(_record(), 5)
     assert prefix == _record()["trajectory"][:4]
 
@@ -82,47 +82,36 @@ def test_boundary_must_be_an_assistant_turn() -> None:
     result = _boundary()
     result["truncate_before_turn"] = 4
     with pytest.raises(PolicyViolation, match="assistant turn"):
-        validate_boundary(_record(), _blocks(), result)
+        validate_boundary(_record(), result)
 
 
-def test_boundary_evidence_must_quote_boundary_turn() -> None:
+def test_boundary_evidence_must_include_boundary_turn() -> None:
     result = _boundary()
-    result["evidence"] = [{"turn_id": 6, "quote": "tests failed"}]
+    result["evidence_turns"] = [6]
     with pytest.raises(PolicyViolation, match="excluded assistant turn"):
-        validate_boundary(_record(), _blocks(), result)
+        validate_boundary(_record(), result)
 
 
-def test_boundary_quote_must_be_grounded() -> None:
+def test_keep_full_cannot_contain_a_boundary() -> None:
-    result = _boundary()
-    result["evidence"][0]["quote"] = "invented evidence"
-    with pytest.raises(PolicyViolation, match="not present"):
-        validate_boundary(_record(), _blocks(), result)
-
-
-def test_keep_full_requires_safe_full_trajectory() -> None:
     result = _boundary()
     result.update(
         decision="KEEP_FULL",
-        candidate_block_id=None,
         truncate_before_turn=None,
-        prefix_safe_before_boundary=True,
         category="NONE",
         severity="NONE",
         state_effect="NONE",
-        evidence=[],
+        evidence_turns=[],
     )
-    validate_boundary(_record(), _blocks(), result)
+    validate_boundary(_record(), result)
-    result["prefix_safe_before_boundary"] = False
+    result["truncate_before_turn"] = 5
-    with pytest.raises(PolicyViolation, match="safe full trajectory"):
+    with pytest.raises(PolicyViolation, match="cannot contain a boundary"):
-        validate_boundary(_record(), _blocks(), result)
+        validate_boundary(_record(), result)
 
 
 def test_prefix_quality_cannot_reference_suffix() -> None:
     prefix = materialize_prefix(_record(), 5)
     result = _quality()
-    result["prefix_valid"] = False
+    result["evidence_turns"] = [5]
-    result["unrecovered_major_or_critical"] = True
-    result["evidence"] = [{"turn_id": 5, "quote": "harmful patch"}]
     with pytest.raises(PolicyViolation, match="outside the prefix"):
         validate_prefix_quality("sample-1", prefix, result)
 
@@ -133,6 +122,48 @@ def test_quality_score_and_tier_are_computed_locally() -> None:
     assert score["quality_tier"] == "MEDIUM"
 
 
+def test_unrecovered_major_issue_is_locally_invalid() -> None:
+    result = _quality()
+    result["behavior_issues"] = [
+        {
+            "turn_id": 3,
+            "kind": "ERROR",
+            "severity": "MAJOR",
+            "recovered": False,
+            "reason": "The prefix leaves a known broken edit.",
+        }
+    ]
+    result["evidence_turns"] = [3]
+    validate_prefix_quality("sample-1", materialize_prefix(_record(), 5), result)
+    assert derive_prefix_safety(result)["prefix_valid"] is False
+    assert compute_prefix_quality(result)["quality_tier"] == "REJECT"
+
+
+@pytest.mark.parametrize("resolved", [0, -1])
+def test_non_success_outcome_is_capped_before_first_stateful_turn(resolved: int) -> None:
+    record = _record()
+    record["resolved"] = resolved
+    record["tools"] = [
+        {"type": "function", "function": {"name": "str_replace_editor"}}
+    ]
+    record["trajectory"][2]["tool_calls"] = [
+        {
+            "id": "call-1",
+            "type": "function",
+            "function": {
+                "name": "str_replace_editor",
+                "arguments": '{"command":"str_replace","path":"src/a.py"}',
+            },
+        }
+    ]
+    result = _boundary()
+    result.update(decision="KEEP_FULL", truncate_before_turn=None)
+    policy = effective_boundary_policy(record, result)
+    assert policy["decision"] == "TRUNCATE"
+    assert policy["truncate_before_turn"] == 3
+    assert policy["source"] == "NON_SUCCESS_FIRST_STATEFUL_CAP"
+
+
 def test_boundary_payload_hides_outcome_and_patch_metadata() -> None:
     record = _record()
     record["resolved"] = -1
@@ -142,7 +173,7 @@ def test_boundary_payload_hides_outcome_and_patch_metadata() -> None:
     }
     serialized = json.dumps(prepare_boundary_payload(record), ensure_ascii=False)
     assert "resolved" not in serialized
-    assert "MODEL_PATCH_SECRET" not in serialized
+    assert "MODEL_PATCH_SECRET" in serialized
     assert "REFERENCE_PATCH_SECRET" not in serialized
 
 

tests/test_audit_workflow.py

@@ -49,20 +49,22 @@ def test_workflow_calls_boundary_then_scores_only_materialized_prefix() -> None:
         [
             {
                 "sample_id": "sample-1",
+                "checks": {
+                    "task_coverage": "INCOMPLETE",
+                    "final_patch_scope": "POLLUTED",
+                    "constraints": "RESPECTED",
+                    "claims_vs_observations": "CONTRADICTED",
+                },
                 "decision": "TRUNCATE",
-                "candidate_block_id": "block-001",
                 "truncate_before_turn": 5,
-                "prefix_safe_before_boundary": True,
                 "category": "PERSISTENT_WRONG_IMPLEMENTATION",
                 "severity": "MAJOR",
                 "state_effect": "UNRECOVERED",
-                "evidence": [{"turn_id": 5, "quote": "BAD_SUFFIX_SENTINEL patch"}],
+                "evidence_turns": [5],
                 "reason": "The patch is not repaired.",
             },
             {
                 "sample_id": "sample-1",
-                "prefix_valid": True,
-                "unrecovered_major_or_critical": False,
                 "behavior_issues": [],
                 "dimensions": {
                     "planning": 15,
@@ -71,7 +73,7 @@ def test_workflow_calls_boundary_then_scores_only_materialized_prefix() -> None:
                     "progress": 12,
                     "clarity_and_efficiency": 10,
                 },
-                "evidence": [],
+                "evidence_turns": [],
                 "reason": "The prefix is useful.",
             },
         ]
@@ -92,14 +94,18 @@ def test_hold_boundary_skips_quality_call() -> None:
         [
             {
                 "sample_id": "sample-1",
+                "checks": {
+                    "task_coverage": "COMPLETE",
+                    "final_patch_scope": "CLEAN",
+                    "constraints": "RESPECTED",
+                    "claims_vs_observations": "CONSISTENT",
+                },
                 "decision": "HOLD",
-                "candidate_block_id": None,
                 "truncate_before_turn": None,
-                "prefix_safe_before_boundary": False,
                 "category": "NONE",
                 "severity": "NONE",
                 "state_effect": "UNCLEAR",
-                "evidence": [],
+                "evidence_turns": [],
                 "reason": "No clear unrepaired defect.",
             }
         ]
@@ -115,25 +121,28 @@ def test_keep_full_process_trajectory_is_scored() -> None:
         [
             {
                 "sample_id": "sample-1",
+                "checks": {
+                    "task_coverage": "COMPLETE",
+                    "final_patch_scope": "CLEAN",
+                    "constraints": "RESPECTED",
+                    "claims_vs_observations": "CONSISTENT",
+                },
                 "decision": "KEEP_FULL",
-                "candidate_block_id": None,
                 "truncate_before_turn": None,
-                "prefix_safe_before_boundary": True,
                 "category": "NONE",
                 "severity": "NONE",
                 "state_effect": "NONE",
-                "evidence": [],
+                "evidence_turns": [],
                 "reason": "No unrepaired severe problem is visible.",
             },
             {
                 "sample_id": "sample-1",
-                "prefix_valid": True,
-                "unrecovered_major_or_critical": False,
                 "behavior_issues": [
                     {
-                        "assistant_turn": 3,
+                        "turn_id": 3,
                         "kind": "INEFFICIENCY",
                         "severity": "MINOR",
+                        "recovered": True,
                         "reason": "The inspection was somewhat broad.",
                     }
                 ],
@@ -144,7 +153,7 @@ def test_keep_full_process_trajectory_is_scored() -> None:
                     "progress": 12,
                     "clarity_and_efficiency": 10,
                 },
-                "evidence": [],
+                "evidence_turns": [],
                 "reason": "Useful despite minor inefficiency.",
             },
         ]
@@ -153,4 +162,4 @@ def test_keep_full_process_trajectory_is_scored() -> None:
     assert len(client.calls) == 2
     assert result["prefix"]["retained_turn_count"] == 6
     assert result["quality"]["local_score"]["issue_counts"]["inefficiencies"] == 1
-    assert result["recommended_use"] == "PROCESS_PREFIX_CANDIDATE"
+    assert result["recommended_use"] == "HOLD"