- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
ReFixAgent explores multiple candidate edit-location sets, generates patches constrained to each set, and progressively selects and refines candidates using trajectory analysis and regression testing. Tasks with multiple distinct localization sets additionally receive graph-based localization information. A final generation stage synthesizes guidance from earlier attempts. On the 500-instance SWE-bench Verified benchmark, the paper reports 74.4% resolution with DeepSeek-V3.2, compared with 69.4% for the strongest reported same-backbone baseline. It also reports improvements with other backbones and when incorporating the mechanism into Mini-SWE-agent and Live-SWE-agent.
Critical strengths
- Exploring alternative localization hypotheses is a sensible way to address repeated repair attempts that remain trapped at the same incorrect edit location. The separation between search over locations and patch refinement makes the proposal understandable and potentially reusable.
- The reported gains appear across several model configurations. The integrations with two existing agent frameworks are useful evidence that the idea is not tied to one implementation.
Critical weaknesses
-
The main comparison does not isolate the proposed search strategy from additional inference and selection. ReFixAgent internally samples multiple localization sets and patches, invokes several analysis or selection agents, and performs a final synthesis. Reporting one returned patch per task is compatible with a system-level pass@1 measure, but does not make this computationally comparable to a single baseline rollout. Appendix A.1 states a 50-agent-step budget without making sufficiently clear whether every agent call, graph exploration, candidate generation, and selection operation is charged to one shared budget, or how equivalent limits are applied to the baselines. No token, cost, or wall-time comparison establishes efficiency. The central experiment should compare against budget-matched independent sampling plus selection, and repeated refinement at one location. Without this control, the gains do not establish that localization diversity is better than allocating similar computation to a simpler search procedure.
-
The proposed mechanism is plausible but insufficiently diagnosed. The paper motivates a lack of search diversity, yet does not quantify how often additional localization sets recover the correct region, how often the selector retains a correct candidate, or whether constrained editing rejects valid repairs outside the initial ranges. This matters because the procedure repeatedly discards half the candidate sets and cannot necessarily recover a discarded strategy. Section 3.2 also filters out existing tests deemed unrelated to the issue, which can remove evidence of regressions elsewhere; the cost-benefit of that filter is not evaluated. A focused analysis of localization diversity, candidate survival, and final selection accuracy would support the mechanism more directly than resolution-only ablations. The graph component improves the score by only 0.2 points on each reported backbone, equivalent to one issue in a single 500-task run, so its claimed contribution needs uncertainty or targeted evidence.
Questions for authors
- What is the complete per-task computation accounting, and how does ReFixAgent compare with independent sampling plus the same selector at an equal budget?
- What exactly is averaged in Table 2, and are additional predicted locations penalized? Can the authors provide counts, seeds, and corrected gain calculations?
- Are the 50 pilot tasks included in the main evaluation, and what is the result after excluding them?
- How frequently does selection discard a viable patch or location, and what happens when a correct repair requires editing outside the allowed ranges?