acceptodds
Under review as a conference paper at ICLR 2027

Are Agents Leaving Your Code Stupid? Predictive Refactoring for Post-Repair Hardening

Abstract

Software engineering agents are evaluated mainly by issue-resolution accuracy, and their workflows are reactive: the agent fixes the reported bug and stops, even when repository history marks the surrounding code as defect-prone. In tangled, high-churn legacy code, a patch can then pass the benchmark while leaving brittle interfaces, recurring defect hotspots, and hidden temporal coupling in place. We propose Memory-Augmented Predictive Refactoring (MAPR), a two-phase workflow that treats a solved repair as the starting point for regression-aware hardening. MAPR builds repository memory from historical pull requests, summarizing bug density, author-linked defect patterns, temporal coupling, review coverage, and code churn. After Phase 1 resolves the issue, Phase 2 replays the Phase 1 patch in a fresh sandbox and hardens the repaired code under a test-driven RED–GREEN–VERIFY protocol that targets guard clauses, diagnostic error messages, and a controlled blast radius. We evaluate MAPR on Phase 1-solved SWE-bench Verified instances with four backbones, together with a 348-instance MiniMax-M2.5 run over the full benchmark. Full Phase 2 raises guard density by a factor of 4.5–7.0 on every backbone and keeps 96.4–100% of repairs on the three backbones with complete harness logs; on the 348-instance run, it raises guard clauses from 68 to 798 while keeping 96.5% of repairs. The RED–GREEN–VERIFY scaffold accounts for most of the gain in hardening quality: removing it lowers the share of error messages that report runtime state on every backbone, for example from 79.7% to 30.4% on MiniMax-M2.5. Separating the two stages also matters: a single-session agent with the same scaffold and step budget loses 6–14 repairs on these backbones, versus 0–2 for MAPR. Falling back to the Phase 1 patch whenever Phase 2 breaks a repair restores every repair and retains at least 97% of the added guards. The anonymized code, data, and analysis artifacts are available at https://figshare.com/s/b1f96ed457df4694a5a4.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, software engineering agents, coding agents, automated program repair, predictive refactoring, repository memory, test-driven development

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6bMYOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Memory-Augmented Predictive Refactoring (MAPR), a two-stage workflow that first repairs a reported issue and then performs localized defensive hardening in a fresh sandbox. The second stage combines repository-history summaries with a RED–GREEN–VERIFY procedure that generates temporary tests, adds defensive checks and informative exceptions, and verifies preservation of the repair. Experiments on SWE-bench Verified report increased defensive-code metrics and high raw repair retention on three out of the four model backbones.

Critical strengths
  1. The paper has a good motivation. The paper observes that large code projects especially those in companies often only fix local codes aimed at quickly recovering the urgent incidents without refining the global code structure and considering long-term stableness.

  2. The paper proposes a good method to extract memory of a repository and control context length at the hardening phase. It extracts file editing history from pull requests and organize the file relationship by associating the target file and the files frequently edited together. When hardening a codebase, it provides the files frequently edited together alongside the repaired files. This is an effective method for identifying a reasonable scope of code context.

Critical weaknesses
  1. Benchmark ground truths enter generation. Section 2.3 and Appendix B.4 explicitly provide the Phase 2 agent with oracle FAIL_TO_PASS and PASS_TO_PASS test names. First, this violates the SWE-bench protocol. When solving SWE-bench instances, even when refining the prediction patches, oracle FAIL_TO_PASS and PASS_TO_PASS should not be provided to the agent. Oracle test names will explicitly help agents to avoid breaking the phase 1 repair code, which cannot reflect the language models’ real capability to prevent breaking the correct fix. Second, in real-world issue resolution, the oracle tests do not exist at all, and agents must write such reproduction tests themselves. Therefore, providing oracle test names to agents is too ideal and cannot reflect realistic issue resolution scenarios.

  2. Oracle tests are used to select agent patches. The paper uses SWE-bench oracle evaluation tests to select the final prediction patch from phase 1 patch and phase 2 patch. Still, same as Weakness 1, this violates the SWE-bench protocol which requires ground truth tests should not be used in any way to improve agent result before patch submission, and using oracle tests to select agent patches is totally meaningless as you can never get such oracles in real-world issue resolution. Then what is the meaning to use ground truths to improve agent success rates?

I think Weakness 1 and 2 are major deficits of the paper. The major contribution of the paper is its phase 2 but I believe the setting of phase 2 is wrong. Therefore, I believe the paper needs major revision to resolve these two issues, and I cannot suggest strong acceptance of the paper.

Questions for authors
  1. The authors should clarify why they intentionally expose ground truth tests in agent input and in prediction patch selection as in real-world issue resolution oracles are not available. If the authors can persuade me, I will increase my rating. But from my point of view, I still suggest the authors changing the experiment setting entirely to:
  • Don’t provide oracle test names to the phase 2 agent;
  • Don’t apply oracle test patch to the codebase in phase 1 and phase 2;
  • Don’t use oracle tests to select the final patch from phase 1 patch and phase 2 patch.

Instead, the authors should:

  • Provide the reproduction tests written by phase 1 agent to the phase 2 agent as the fix preservation guard, and
  • Use the reproduction tests written by phase 1 agent to select between phase 1 patch and phase 2 patch.

This setting can reflect language models’ realistic capabilities to harden a code repair.

  1. Experiments only on the old SWE-bench Verified are not convincing enough now. Add evaluation on newer, multilingual SWE benchmarks such as SWE-bench-Live/MultiLang or ScaleAI/SWE-bench_Pro (test split), at least for two language models.

  2. As Claude is the state-of-the-art model provider for SWE tasks, try to add at least one Claude-Sonnet model in your evaluation.

Reviewer_8CX7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAPR, a two-phase workflow that treats a solved repair as the starting point for regression-aware hardening. The results is impressive: full Phase 2 raises guard density by 4.5--7.0 on every backbone and keeps 96.4--100% of repairs on the three backbones.

Critical strengths

Identify a critical problem that may affect the deployment of well-scored coding agents.

Critical weaknesses
  1. The authors acknowledge that the historical information contains clear temporal leakage, yet the main experiments are not rerun with a clean history. This means the reported performance gains may partly reflect time-correlated artifacts rather than the intended signal, making it difficult to trust the paper's central empirical claims.
  2. The work's value depends on models currently having weak test-writing abilities; as this capability improves, the approach loses its motivation. Compounding this, Memory—advertised as a core component in the title—is not convincingly justified by ablation: removing Memory increases MiniMax's guard count from 137 to 148, and safety gains are inconsistent across models. Together, these issues leave the paper's key contributions under-substantiated.
Questions for authors
  1. The writing and figure quality are not high, and the paper is rather difficult to follow.
  2. Memory is a core component highlighted in the title, yet the ablation study does not convincingly demonstrate its necessity. In Table 2, after removing memory: MiniMax's guard count increases from 137 to 148, while Qwen's changes from 258 to 255. The safety gains are also inconsistent: some models show fewer regression issues, whereas others show no improvement.
  3. The authors acknowledge that the historical information suffers from clear temporal leakage, yet the main experiments are not rerun with a clean history. This is a fairly central issue; without doing so, it is difficult to fully substantiate the paper's claims.
  4. My understanding is that if models' test-writing capabilities improve, the value of this work diminishes—that is, this work is only applicable under the condition that models currently have weak test-writing abilities.
Reviewer_6sh2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adds a second step after code repair to make the code more robust through input checks and clearer error messages. This step uses the repaired patch, repository history, and a RED-GREEN-VERIFY process guided by agent-written temporary tests. If the new patch fails the benchmark tests, a fallback restores the first-stage repair. The approach is evaluated on SWE-bench Verified with models including MiniMax-M2.5, Gemini-3.1-Flash-Lite, Qwen3.5-Plus, and GPT-5-mini. The results show more defensive checks, and ablations examine the roles of history and the test-writing process.

Critical strengths
  • Making repaired code more robust is an important and relevant problem. Treating additional defensive edits as a separate, optional step is a practical idea.

  • The study includes several model families, a larger MiniMax run, and ablations that remove history or the test-writing scaffold. These help examine the components separately.

  • The test-driven process gives the agent a concrete way to check its proposed edits rather than adding defensive code without testing it.

  • Metrics such as guard density make the changes in patch structure measurable. Using repository history to guide these edits is also an interesting idea, although its measured contribution is limited.

Critical weaknesses
  • My main concern is whether the added checks actually improve the code. Guard counts and error-message features measure what the prompt asks the agent to add, not whether future bugs become less likely. Existing benchmark tests and the agent's own temporary tests may miss strict checks or valid inputs that are newly rejected. The evaluation therefore does not directly measure the usefulness of Phase 2.

  • The reported fallback uses benchmark hidden-test results to decide when to restore the Phase 1 patch. In deployment, those tests would not necessarily be available. The paper notes that repository tests could be used instead, but does not show whether they would detect the same regressions. Since the raw Phase 2 outputs do lose previously passing repairs, this distinction matters. A fallback evaluated using only deployment-available tests, with hidden tests reserved for final evaluation, would provide stronger evidence of practical safety.

  • Repository memory is presented as a key technical component, but the ablations show little consistent benefit from it. It is unclear whether memory changes the kinds of checks added or their usefulness, even when aggregate guard counts remain similar. Paired examples with and without memory would help establish whether this component provides meaningful guidance.

  • A smaller concern is the single-stage comparison. A better control would continue from the same passing repair checkpoint, with and without a fresh session. Otherwise, some single-stage failures may reflect failure to find the initial repair rather than hardening breaking an existing repair. Please also clarify whether both settings receive the same hidden test-based fallback.

Questions for authors
  • How are the new guards checked to ensure that they accept valid inputs beyond those covered by the existing tests?

  • Why does repository history have a limited measured effect? Are there qualitative differences in the relevance or usefulness of the checks added with and without memory?

  • What explains GPT-5-mini's higher regression rate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.