- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Memory-Augmented Predictive Refactoring (MAPR), a two-stage workflow that first repairs a reported issue and then performs localized defensive hardening in a fresh sandbox. The second stage combines repository-history summaries with a RED–GREEN–VERIFY procedure that generates temporary tests, adds defensive checks and informative exceptions, and verifies preservation of the repair. Experiments on SWE-bench Verified report increased defensive-code metrics and high raw repair retention on three out of the four model backbones.
Critical strengths
-
The paper has a good motivation. The paper observes that large code projects especially those in companies often only fix local codes aimed at quickly recovering the urgent incidents without refining the global code structure and considering long-term stableness.
-
The paper proposes a good method to extract memory of a repository and control context length at the hardening phase. It extracts file editing history from pull requests and organize the file relationship by associating the target file and the files frequently edited together. When hardening a codebase, it provides the files frequently edited together alongside the repaired files. This is an effective method for identifying a reasonable scope of code context.
Critical weaknesses
-
Benchmark ground truths enter generation. Section 2.3 and Appendix B.4 explicitly provide the Phase 2 agent with oracle
FAIL_TO_PASSandPASS_TO_PASStest names. First, this violates the SWE-bench protocol. When solving SWE-bench instances, even when refining the prediction patches, oracleFAIL_TO_PASSandPASS_TO_PASSshould not be provided to the agent. Oracle test names will explicitly help agents to avoid breaking the phase 1 repair code, which cannot reflect the language models’ real capability to prevent breaking the correct fix. Second, in real-world issue resolution, the oracle tests do not exist at all, and agents must write such reproduction tests themselves. Therefore, providing oracle test names to agents is too ideal and cannot reflect realistic issue resolution scenarios. -
Oracle tests are used to select agent patches. The paper uses SWE-bench oracle evaluation tests to select the final prediction patch from phase 1 patch and phase 2 patch. Still, same as Weakness 1, this violates the SWE-bench protocol which requires ground truth tests should not be used in any way to improve agent result before patch submission, and using oracle tests to select agent patches is totally meaningless as you can never get such oracles in real-world issue resolution. Then what is the meaning to use ground truths to improve agent success rates?
I think Weakness 1 and 2 are major deficits of the paper. The major contribution of the paper is its phase 2 but I believe the setting of phase 2 is wrong. Therefore, I believe the paper needs major revision to resolve these two issues, and I cannot suggest strong acceptance of the paper.
Questions for authors
- The authors should clarify why they intentionally expose ground truth tests in agent input and in prediction patch selection as in real-world issue resolution oracles are not available. If the authors can persuade me, I will increase my rating. But from my point of view, I still suggest the authors changing the experiment setting entirely to:
- Don’t provide oracle test names to the phase 2 agent;
- Don’t apply oracle test patch to the codebase in phase 1 and phase 2;
- Don’t use oracle tests to select the final patch from phase 1 patch and phase 2 patch.
Instead, the authors should:
- Provide the reproduction tests written by phase 1 agent to the phase 2 agent as the fix preservation guard, and
- Use the reproduction tests written by phase 1 agent to select between phase 1 patch and phase 2 patch.
This setting can reflect language models’ realistic capabilities to harden a code repair.
-
Experiments only on the old
SWE-bench Verifiedare not convincing enough now. Add evaluation on newer, multilingual SWE benchmarks such asSWE-bench-Live/MultiLangorScaleAI/SWE-bench_Pro (test split), at least for two language models. -
As Claude is the state-of-the-art model provider for SWE tasks, try to add at least one
Claude-Sonnetmodel in your evaluation.