- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how coding agents should reuse historical repair experience when an old repair is related to the current issue but its original patch is no longer a valid solution. It proposes a benchmark called RepairMem with controlled (or called constructed) and real "historical and current repair pairs". And it also proposes ReGround, which reassesses the old patch on the current task and provides current code evidence before the agent begins repair. Across 5 models, ReGround improves repair success and reduces model calls compared with no memory condition and direct reuse of historical repairs.
Critical strengths
-
The problem is practical and clearly motivated and the method is simple and intuitive.
-
The benchmark includes both controlled (or called constructed) and real constructions repair pairs, and the evaluation covers 5 coding models.
-
Most of the presentation of this paper is smooth and easy to follow, the content is well structured.
Critical weaknesses
-
In the Introduction, the paragraph "To evaluate this reuse directly, we introduce REPAIRMEM xxx" suddenly introduces two concepts, which makes the presentation very hard to follow. Controlled tasks are easy to understand, but particularly I can't get what "naturally sourced tasks" means, and I have to read the later sections to roughly understand it. Why not directly call them constructed tasks and real tasks, or use names that are easier to understand?
-
The main text does not clearly define what is a historical repair experience or how a related repair would be identified in a real memory system. In the benchmark, each task is directly provided with a selected historical repair record and patch. Controlled task's relation is clear since it is based on specific code relation construction, but for natural task, it is unclear what memories would be stored in practice and what method would retrieve or select the "related" repair for a current task.
-
The main experiment of the comparison does not isolate the effect of regrounding the historical repair. ReGround receives current execution feedback and relevant source excerpts, while No Memory and Raw History do not. According to my understanding, a normal coding agent should also execute code or obtain some relevant excerpts.
-
In addition, the claim that "raw historical repair can reduce repair success" is not very rigorous. All agents operate under a fixed 24 call budget (intuitively, I feel this may be too small, because using historical experience for testing is a very normal thing to do), and the matched-evidence results suggest that a major effect of ReGround is reducing interaction cost. It is therefore unclear whether Raw History is intrinsically harmful or mainly less efficient under the current budget.
-
The construction of the Controlled tasks is not clearly explained, including how the current faults are generated, what types of faults are included, how the final tasks are selected, and whether the constructed cases receive any independent or human validation.
If the authors can address the weaknesses and questions I mentioned, I would be happy to raise my rating to 3.
Questions for authors
- How would a "related" historical repair be retrieved in a real memory system? And what is the definition of "related"?
- Does Raw History still hurt with a larger interaction budget?
- How are the Controlled-task faults generated and selected?
- Are the constructed Controlled cases independently or manually validated? What are him/her/them background? SE expert? SE major Ph.D. student?
- Also address the concerns I mentioned in Weaknesses part.