acceptodds
Withdrawn from ICLR 2027

SAGA-Edit: Tracking Natural-Language Obligations to Reduce Unauthorized Drift in Repository-Level Code Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Repository-level code edits can solve an issue while changing behavior the task did not authorize. We study this failure as drift across accepted editing transitions. SAGA-EDIT tracks evidence-backed fix and preserve obligations, checks candidate changes against them, and records decisions in a persistent ledger. On a 104-task subset, the full controller solves tasks, with drift in trajectories, compared with tasks solved and with drift without the ledger. Across 168 tasks, full evidence yields solved tasks and with drift, versus and with code/test/API evidence alone. In a 12-outcome exploratory family, the corresponding code-only drift contrast has Holm-adjusted ; no-ledger and final-state-only drift contrasts have adjusted and . A 120-task public-set comparison does not establish solve noninferiority within a 2-percentage-point margin. These results support tracking obligations through a trajectory, while incomplete specifications and limited test coverage preclude a general guarantee of semantic preservation.

Primary area:
datasets and benchmarks
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xp3nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies unauthorized semantic drift in repository-level code editing: a coding agent may solve the requested issue while changing behavior that the task did not authorize. SAGA-EDIT addresses this by extracting fix and preserve obligations from task/repository/review evidence, tracking them across editing rounds in a ledger, and checking candidate patches for authorized progress, preservation loss, and unattributed semantic changes. The main empirical claim is that obligation tracking, especially with full natural-language evidence and persistent ledger state, reduces observed unauthorized drift on the authors’ benchmark.

Critical strengths
  • The paper studies an important failure mode of repository-level code editing: accepted patches may pass visible tests while changing behavior not authorized by the task.
  • The method explicitly tracks fix and preserve obligations across multi-round editing trajectories.
  • The experiments include comprehensive ablations such as no-ledger, final-state-only, and code/test/API-only evidence.
Critical weaknesses
  • The novelty over regression testing and oracle construction is not fully clear. The method can be viewed as obligation-guided regression testing: natural-language evidence or human criteria define which old behaviors should be preserved, and generated probes/checks test whether candidate patches preserve them. The two central challenges addressed by the paper—insufficient regression coverage and uncertainty about which behavior changes are intended—are standard regression-testing/oracle problems. The paper should better isolate whether the improvement comes from a new semantic governance mechanism or from adding more targeted regression criteria/tests.

  • The drift evaluation depends heavily on human oracle anchors, which are necessarily incomplete and manually constructed. Therefore, the reported metric measures observed drift relative to annotated obligations, not all possible unauthorized semantic changes. This also raises scalability concerns: if high-quality human annotation is required to define preserve obligations, it may be expensive and difficult to scale to larger repositories or long-lived real-world projects.

  • More details are needed about the annotation protocol: number of annotators, whether they were blinded to system outputs, inter-annotator agreement, disagreement resolution, and whether oracle anchors were created before or after inspecting candidate patches. Without this, it is hard to assess subjectivity, hindsight bias, or leakage.

  • The public-task set result is relatively weak because the benchmark identity/version is not documented and the outcome is only a regression-test proxy rather than the same oracle-drift measure used for SB1/SB3.

Questions for authors
  1. How is SAGA-EDIT conceptually different from an evolving regression-test oracle derived from issue/review/documentation evidence?
  2. Can the authors compare against a baseline that directly generates preservation/regression tests from the same evidence?
  3. How were human oracle anchors constructed? Were they defined before inspecting candidate patches/system outputs, were annotators blinded, and what was the inter-annotator agreement?
  4. Were oracle anchors defined before inspecting candidate patches/system outputs?
Reviewer_MHLvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Repo-level code edits from coding agents can have side effects that change behavior in later agent runs. To mitigate undesired effects, the paper introduces SAGA-EDIT, a method that tracks natural language requirements across edits in a obligation ledger. The ledger distinguishes required fixes from behaviors to preserve and updates which obligations remain active when later instructions authorize changes. In the 168-task evaluation, SAGA-EDIT records behavioral drift on 11 tasks, compared with 26 for a variant using only code, test, and API evidence.

Critical strengths
  1. The paper provides an interesting and useful problem formulation in practice. Tracking which requirements remain active across successive edits addresses a practical need in repository level coding agents.
  2. The design of SAGA-EDIT is model and harness independent at the architectural level, which can be useful. Its placement between candidate generation and patch acceptance makes obligation tracking potentially reusable across coding agent workflows.
Critical weaknesses
  1. The paper is poorly presented in general. In the abstract, much of task counts and adjusted p-values make the central idea difficult to follow. A clearer explanation of how the controller uses its ledger would help readers interpret these results.
  2. The evaluation relies heavily on internal controller variants as baselines. Comparisons with current coding agents under same generation and verification budgets would better show the method’s practical value.
  3. The ablation shows that natural-language evidence provides the strongest statistical support for drift reduction (Table 2). The ledger contrast has Holm-adjusted p = .08124, so a control retaining the same language history would help isolate the contribution of explicit obligation states.
Questions for authors
  1. How does SAGA-EDIT compare with established coding agents under matched generation and verification budgets?
  2. What evidence isolates the benefit of explicit obligation states when both SAGA-EDIT and the baseline retain the same natural-language history?
Reviewer_m7zsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SAGA-EDIT, a controller for repository-level code editing agents that sits between candidate generation and patch acceptance. It extracts evidence-backed obligations from the task, review threads, documentation, and code. 'Fix' obligations describe behavior the task asks the change and 'preserve' obligations describe behavior that must remain.

For each candidate patch, the controller attributes the behavioral changes to these obligations and checks them with generated tests, static analysis, mutation probes, and entailment. Its decisions including explicit supersession of an obligation by a later review are kept in a persistent ledger.

The evaluation measures solving the task separately from unauthorized drift and consists of internal ablations on several task sets. On the 104-task subset, the full controller solves 74 tasks and drifts on 6. Without the ledger, it solves 68 and drifts on 17.

Critical strengths
  1. The paper identifies a real and under-measured problem.

  2. The statistical analysis is careful and honest. The paper uses paired tests with multiple-comparison correction and reports non-significant results as such.

Critical weaknesses
  1. The results cannot be interpreted because basic information about the experiments is missing. The paper does not mention the language model used, how the benchmarks were constructed, where the public task set comes from, and how the obligation and drift labels were annotated.

  2. Therefore, the reported numbers cannot be checked, reproduced, or compared with other work just from the given information.

  3. The evaluation does not establish that the idea works better than simple alternatives. All comparisons are against ablated versions of the proposed controller. There is no standard benchmark and no existing agent with an ordinary regression test, which is the obvious baseline.

Questions for authors
  1. Which language model or models were used at each stage?

  2. How were the task sets constructed, and how were the obligation and drift labels annotated?

  3. How does SAGA-EDIT compare, on a recognized benchmark and at what cost, with an existing agent that accepts patches based on a standard regression-test check?

  4. How was the candidate cache built, and how many branches per variant ended because a cache entry was missing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.