- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies unauthorized semantic drift in repository-level code editing: a coding agent may solve the requested issue while changing behavior that the task did not authorize. SAGA-EDIT addresses this by extracting fix and preserve obligations from task/repository/review evidence, tracking them across editing rounds in a ledger, and checking candidate patches for authorized progress, preservation loss, and unattributed semantic changes. The main empirical claim is that obligation tracking, especially with full natural-language evidence and persistent ledger state, reduces observed unauthorized drift on the authors’ benchmark.
Critical strengths
- The paper studies an important failure mode of repository-level code editing: accepted patches may pass visible tests while changing behavior not authorized by the task.
- The method explicitly tracks fix and preserve obligations across multi-round editing trajectories.
- The experiments include comprehensive ablations such as no-ledger, final-state-only, and code/test/API-only evidence.
Critical weaknesses
-
The novelty over regression testing and oracle construction is not fully clear. The method can be viewed as obligation-guided regression testing: natural-language evidence or human criteria define which old behaviors should be preserved, and generated probes/checks test whether candidate patches preserve them. The two central challenges addressed by the paper—insufficient regression coverage and uncertainty about which behavior changes are intended—are standard regression-testing/oracle problems. The paper should better isolate whether the improvement comes from a new semantic governance mechanism or from adding more targeted regression criteria/tests.
-
The drift evaluation depends heavily on human oracle anchors, which are necessarily incomplete and manually constructed. Therefore, the reported metric measures observed drift relative to annotated obligations, not all possible unauthorized semantic changes. This also raises scalability concerns: if high-quality human annotation is required to define preserve obligations, it may be expensive and difficult to scale to larger repositories or long-lived real-world projects.
-
More details are needed about the annotation protocol: number of annotators, whether they were blinded to system outputs, inter-annotator agreement, disagreement resolution, and whether oracle anchors were created before or after inspecting candidate patches. Without this, it is hard to assess subjectivity, hindsight bias, or leakage.
-
The public-task set result is relatively weak because the benchmark identity/version is not documented and the outcome is only a regression-test proxy rather than the same oracle-drift measure used for SB1/SB3.
Questions for authors
- How is SAGA-EDIT conceptually different from an evolving regression-test oracle derived from issue/review/documentation evidence?
- Can the authors compare against a baseline that directly generates preservation/regression tests from the same evidence?
- How were human oracle anchors constructed? Were they defined before inspecting candidate patches/system outputs, were annotators blinded, and what was the inter-annotator agreement?
- Were oracle anchors defined before inspecting candidate patches/system outputs?