- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this paper, the authors discuss this core issue: when benchmarking AI models, the evaluation numbers can be affected by the evaluation procedure and protocols. Specifically, three protocols are investigated: (1) P1, independent sampling, each sampling within K attempts are standalone; (2) P2, generic refinement, only failure itself is given to the next attempt; (3) P3, structured test-guided refinement, detailed unit test failure feedbacks are provided to the next attempt. The information “leaked” to the model for a new attempt gradually increases from protocols P1 to P3; it may seem fair to just compare models while fixing the protocol to any one of the three, however, the sensitivity to protocols can diverge significantly among different models.
Critical strengths
S1. Positive appeal to the eval community. This paper acts as a reminder of the phenomenon: eval protocols being under-specified and that many numbers are not directly comparable; it urges the community to report eval results while mentioning the precise protocol used during evaluation.
Critical weaknesses
-
W1 (major). Misleading contributions. The paper lists three contributions, but C1 seems to be the methodology itself rather than contribution, C2 is more of an observation, C3 is confusing as it sounds like proposing a real-world benchmark (which is not). From what I read, the main contribution of the paper is its insight obtained from and verified by the experiments.
-
W2 (major). Bold and flawed claim. The main claim/insight is analogous to stating that when evaluating an AI agent, the harness plays a crucial role in the evaluation numbers. It is true, although not surprising in October 2026; this might already be widely perceived in the AI community. The uniqueness of the paper is to investigate this in the narrowed-down context of coding models for program repair, a particular niche where the evaluation protocol itself acts as an intervention due to the potential information fed back to the models. To make this insight valuable and significant in this context, the paper attempts to prove that evaluating models with the same canonical protocol is not adequate because the sensitivity to the three listed protocols can diverge significantly among different models; the main backup claim is, P3 with structured feedback gives better performance than P1 & P2 on synthetic tasks, but not on real-world tasks (BugsinPy). This is a bold and flawed claim, as in reality, strong coding agents such as claude code with opus5.5 rely heavily on the feedbacks from a failed attempt for debugging. This claim is likely due to the flawed experiment setting.
-
W3 (major). Flawed experiment setting. First, to support the bold claim that structured feedback showing no clear advantage over repeated sampling without feedback, it is expected that consistency across several benchmarks is shown. BugsInPy (work from 2020, 1st line of page 4) is the only datapoint in the paper, I suggest adding at least one more, ideally adding some recent ones (e.g., SWE-bench-Live). Secondly, in this paper's BugsInPy settings, reasoning is explicitly disabled. This directly weakens the model's capability in processing the feedback information of debugging. It explains why P3 does not show advantages, as models are deprived of the abilities to do so. Also, 75.9% of the failures (last paragraph of page 6) are not producing structured feedback as they did not reach that step; this means that in this experiment setting, there is no such feedback information available for P3, not that the protocol of P3 itself showing zero advantages over P1/P2.
Questions for authors
Q1. Related to W1. Could the authors explicitly clarify the contributions of this paper? what are the core conceptual or scientific contributions of this work? C1 seems to be the methodology, C3 could be misleading.
Q2. Related to W2 & W3. In section 7, reasoning was explicitly disabled. is it the case that P3 not showing advantage on BugsInPy is because models are unable to process error logs well without thinking tokens? (combined with the fact that majority of instances have no feedback at all?) Could you provide a clean ablation experiment to verify? (specifically, eval P3 vs P1, strictly on the subset that generated structured test feedback, with reasoning enabled)
Q3. Related to W2 & W3. To support the claim, do the authors have preliminary results or analysis on some recent, ideally widely-recognized repo-level benchmark (under standard coding agent settings)?
Additional feedback (minor, does not affect my rating): When introducing the topic and scope in the intro section, no existing work is mentioned. It might be better if the research question is introduced while citing related research and their connections to the problem discussed in the paper, unless this topic is brand new.
Response to questions and concerns
We thank the reviewers for their feedback. We revised the paper as follows:
New primary SWE-bench Verified experiment (Section 5, Appendix A). We ran 290 repository-stratified instances with Claude Sonnet 5 (adaptive thinking), comparing P1 and P3 at K=3 with a shared first attempt and official SWE-bench grading. P3 raises resolution from 72.8% to 79.3% (+6.6 pp, 95% CI [2.7, 10.4], exact McNemar p = 0.0013).
BenchCraft has become supporting evidence (Appendix C); the main claims rely on two established real code benchmarks.
BugsInPy rerun with reasoning on/off (Section 6, Tables 3–4): 3 models × 2 settings × 2 replicates. P3 > P2 > P1 in both settings; no contrast is conclusive.
Failure-stage analysis (Tables 2, 5; Appendix B) identifies where P3 helps.
Positioning. We discuss Chen et al. (2024) and Olausson et al. (2024) in the Introduction and Related Work and revise the contributions.
Reproducibility. Appendices A–C provide full prompt templates, feedback formats, sampling/reasoning settings, patch-application rules, and statistical formulas.
Answers to Questions:
W1 / Q1 — Contributions are misleading.
We agree and rewrote them (Section 1). There are now two contributions, both empirical. The former C1 (the methodology) and C3 (the benchmark-like phrasing) are removed.
W2 — "Bold and flawed claim" that feedback shows no advantage. We take the point, and the revision no longer makes that claim. With 290 SWE-bench Verified instances added, structured feedback does help significantly (+6.6 pp; +15.4 pp among first-attempt failures). This agrees with the reviewer's intuition from coding agents. Our claim is narrower: the size of the protocol effect depends on the benchmark and on where failures occur. It is significant on SWE-bench and not resolved on BugsInPy (Section 7, "Scope, not contradiction"). A score reported without its protocol is therefore under-specified. We agree that harness sensitivity is broadly recognized. Our contribution is to quantify it for program repair under matched attempt budgets, with paired and clustered uncertainty, and to show where it arises.
W3 / Q2 — Reasoning disabled; most failures produce no structured feedback. Ablation: P3 vs P1 on the subset with test feedback, with reasoning enabled.
We ran exactly this analysis: • BugsInPy, reasoning enabled, test feedback available after attempt 1 (24 units, 11 defects): P3 6/24 vs P1 4/24, +8.3 pp (CI [−10.0, +30.4], McNemar p = 0.69). The direction matches SWE-bench, but the subgroup is too small to be conclusive (Appendix B). • SWE-bench Verified, adaptive thinking enabled, first patch applied but failed the tests (n = 70): P3 54.3% vs P1 32.9%, +21.4 pp (CI [9.6, 33.2], p = 0.0015) (Table 2). • Over the full BugsInPy set with reasoning enabled, the P3−P1 difference is +4.2 pp (Table 3). This is close to the reasoning-disabled value of +5.8 pp (Table 4), so disabling reasoning does not explain the BugsInPy result. We agree with the reviewer's reading. On BugsInPy, the 62–74% of failures that end at patch application mean P3 often has no test feedback to give. The inconclusive outcome reflects limited feedback availability and precision, not evidence that the protocol has no effect. Section 7 now states this directly.
W3 / Q3 — More benchmarks; recent, widely recognized repo-level benchmarks under standard coding-agent settings. We added SWE-bench Verified (290 instances, official harness). We did not run SWE-bench-Live or a full autonomous agent. Our object of study is a controlled, fixed three-attempt protocol, which lets us vary only the information returned between attempts. An agent harness changes many factors at once, including tool use, exploration and self-generated tests. Section 8 states this scope limit, and we view agent-level protocol studies as important future work. One caveat also needs stating here: both prompts include an oracle-derived line range (Section 8), so our setting is easier than standard agent evaluation.
Minor — Related work in the Introduction. Fixed. The Introduction now cites prompt and inference-setting sensitivity work (Lu et al., 2022; Shao et al., 2026) and self-repair work (Chen et al., 2024; Olausson et al., 2024).