acceptodds
Under review as a conference paper at ICLR 2027

Evaluation Is an Intervention: How Evaluation Protocols Shape Measured LLM Program-Repair Performance

Abstract

Benchmark scores are often used to measure a large language model’s capability. We argue that the resulting number can be affected by the evaluation procedure as it determines the opportunities and information available to the model. In this paper, we investigate how evaluation protocol choice changes measured program-repair performance. The effects of repeated sampling without feedback, refinement with generic feedback and refinement with structured feedback are compared under matched maximum attempt budgets. Experiments on the synthetic BenchCraft benchmark show substantial protocol effects across three models. The estimated improvement in visible-test success with structured feedback over repeated sampling without feedback is about 11 percentage points for Claude Haiku, 23 for GPT-5.6-terra and 31 for Claude Sonnet 5. Each protocol allowed up to three attempts per task. The results support a substantial improvement, although uncertainty about its size exceeds the pre-specified precision target. In the real-world BugsInPy benchmark experiment, refinement with structured feedback shows no clear advantage over repeated sampling without feedback at a comparable attempt budget. In this experiment, additional attempts produced successful repairs even without feedback. Success rates also varied substantially across defects. The structured-feedback advantage observed on BenchCraft therefore does not clearly extend to BugsInPy under the execution conditions studied. This leads us to conclude that protocol effects depend on the tasks and execution conditions under which they are measured. Evaluation protocol should therefore be treated as a consequential experimental factor. An LLM benchmark score is under-specified without the protocol and execution regime that produced it.

Primary area:
datasets and benchmarks
Keywords:
Large language models, automated program repair, evaluation protocols, benchmark evaluation, test feedback, iterative refinement, repeated sampling, empirical software engineering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zsRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors discuss this core issue: when benchmarking AI models, the evaluation numbers can be affected by the evaluation procedure and protocols. Specifically, three protocols are investigated: (1) P1, independent sampling, each sampling within K attempts are standalone; (2) P2, generic refinement, only failure itself is given to the next attempt; (3) P3, structured test-guided refinement, detailed unit test failure feedbacks are provided to the next attempt. The information “leaked” to the model for a new attempt gradually increases from protocols P1 to P3; it may seem fair to just compare models while fixing the protocol to any one of the three, however, the sensitivity to protocols can diverge significantly among different models.

Critical strengths

S1. Positive appeal to the eval community. This paper acts as a reminder of the phenomenon: eval protocols being under-specified and that many numbers are not directly comparable; it urges the community to report eval results while mentioning the precise protocol used during evaluation.

Critical weaknesses
  • W1 (major). Misleading contributions. The paper lists three contributions, but C1 seems to be the methodology itself rather than contribution, C2 is more of an observation, C3 is confusing as it sounds like proposing a real-world benchmark (which is not). From what I read, the main contribution of the paper is its insight obtained from and verified by the experiments.

  • W2 (major). Bold and flawed claim. The main claim/insight is analogous to stating that when evaluating an AI agent, the harness plays a crucial role in the evaluation numbers. It is true, although not surprising in October 2026; this might already be widely perceived in the AI community. The uniqueness of the paper is to investigate this in the narrowed-down context of coding models for program repair, a particular niche where the evaluation protocol itself acts as an intervention due to the potential information fed back to the models. To make this insight valuable and significant in this context, the paper attempts to prove that evaluating models with the same canonical protocol is not adequate because the sensitivity to the three listed protocols can diverge significantly among different models; the main backup claim is, P3 with structured feedback gives better performance than P1 & P2 on synthetic tasks, but not on real-world tasks (BugsinPy). This is a bold and flawed claim, as in reality, strong coding agents such as claude code with opus5.5 rely heavily on the feedbacks from a failed attempt for debugging. This claim is likely due to the flawed experiment setting.

  • W3 (major). Flawed experiment setting. First, to support the bold claim that structured feedback showing no clear advantage over repeated sampling without feedback, it is expected that consistency across several benchmarks is shown. BugsInPy (work from 2020, 1st line of page 4) is the only datapoint in the paper, I suggest adding at least one more, ideally adding some recent ones (e.g., SWE-bench-Live). Secondly, in this paper's BugsInPy settings, reasoning is explicitly disabled. This directly weakens the model's capability in processing the feedback information of debugging. It explains why P3 does not show advantages, as models are deprived of the abilities to do so. Also, 75.9% of the failures (last paragraph of page 6) are not producing structured feedback as they did not reach that step; this means that in this experiment setting, there is no such feedback information available for P3, not that the protocol of P3 itself showing zero advantages over P1/P2.

Questions for authors

Q1. Related to W1. Could the authors explicitly clarify the contributions of this paper? what are the core conceptual or scientific contributions of this work? C1 seems to be the methodology, C3 could be misleading.

Q2. Related to W2 & W3. In section 7, reasoning was explicitly disabled. is it the case that P3 not showing advantage on BugsInPy is because models are unable to process error logs well without thinking tokens? (combined with the fact that majority of instances have no feedback at all?) Could you provide a clean ablation experiment to verify? (specifically, eval P3 vs P1, strictly on the subset that generated structured test feedback, with reasoning enabled)

Q3. Related to W2 & W3. To support the claim, do the authors have preliminary results or analysis on some recent, ideally widely-recognized repo-level benchmark (under standard coding agent settings)?

Additional feedback (minor, does not affect my rating): When introducing the topic and scope in the intro section, no existing work is mentioned. It might be better if the research question is introduced while citing related research and their connections to the problem discussed in the paper, unless this topic is brand new.

1 comment
AuthorsOpenReview ↗

Response to questions and concerns

We thank the reviewers for their feedback. We revised the paper as follows:

  1. New primary SWE-bench Verified experiment (Section 5, Appendix A). We ran 290 repository-stratified instances with Claude Sonnet 5 (adaptive thinking), comparing P1 and P3 at K=3 with a shared first attempt and official SWE-bench grading. P3 raises resolution from 72.8% to 79.3% (+6.6 pp, 95% CI [2.7, 10.4], exact McNemar p = 0.0013).

  2. BenchCraft has become supporting evidence (Appendix C); the main claims rely on two established real code benchmarks.

  3. BugsInPy rerun with reasoning on/off (Section 6, Tables 3–4): 3 models × 2 settings × 2 replicates. P3 > P2 > P1 in both settings; no contrast is conclusive.

  4. Failure-stage analysis (Tables 2, 5; Appendix B) identifies where P3 helps.

  5. Positioning. We discuss Chen et al. (2024) and Olausson et al. (2024) in the Introduction and Related Work and revise the contributions.

  6. Reproducibility. Appendices A–C provide full prompt templates, feedback formats, sampling/reasoning settings, patch-application rules, and statistical formulas.

Answers to Questions:

W1 / Q1 — Contributions are misleading.

We agree and rewrote them (Section 1). There are now two contributions, both empirical. The former C1 (the methodology) and C3 (the benchmark-like phrasing) are removed.

  1. Protocol effects on real repository defects. P3 significantly raises SWE-bench Verified resolution over P1 (+6.6 pp). The gain on BugsInPy is similar in direction and size (+5.0 pp) but inconclusive.
  2. Failure conditions associated with refinement effectiveness. The advantage is largest when the initial patch applies but fails the tests. This analysis is exploratory and not causal. The resutls are promising and we're planning to investigate this in a future publication.

W2 — "Bold and flawed claim" that feedback shows no advantage. We take the point, and the revision no longer makes that claim. With 290 SWE-bench Verified instances added, structured feedback does help significantly (+6.6 pp; +15.4 pp among first-attempt failures). This agrees with the reviewer's intuition from coding agents. Our claim is narrower: the size of the protocol effect depends on the benchmark and on where failures occur. It is significant on SWE-bench and not resolved on BugsInPy (Section 7, "Scope, not contradiction"). A score reported without its protocol is therefore under-specified. We agree that harness sensitivity is broadly recognized. Our contribution is to quantify it for program repair under matched attempt budgets, with paired and clustered uncertainty, and to show where it arises.

W3 / Q2 — Reasoning disabled; most failures produce no structured feedback. Ablation: P3 vs P1 on the subset with test feedback, with reasoning enabled.

We ran exactly this analysis: • BugsInPy, reasoning enabled, test feedback available after attempt 1 (24 units, 11 defects): P3 6/24 vs P1 4/24, +8.3 pp (CI [−10.0, +30.4], McNemar p = 0.69). The direction matches SWE-bench, but the subgroup is too small to be conclusive (Appendix B). • SWE-bench Verified, adaptive thinking enabled, first patch applied but failed the tests (n = 70): P3 54.3% vs P1 32.9%, +21.4 pp (CI [9.6, 33.2], p = 0.0015) (Table 2). • Over the full BugsInPy set with reasoning enabled, the P3−P1 difference is +4.2 pp (Table 3). This is close to the reasoning-disabled value of +5.8 pp (Table 4), so disabling reasoning does not explain the BugsInPy result. We agree with the reviewer's reading. On BugsInPy, the 62–74% of failures that end at patch application mean P3 often has no test feedback to give. The inconclusive outcome reflects limited feedback availability and precision, not evidence that the protocol has no effect. Section 7 now states this directly.

W3 / Q3 — More benchmarks; recent, widely recognized repo-level benchmarks under standard coding-agent settings. We added SWE-bench Verified (290 instances, official harness). We did not run SWE-bench-Live or a full autonomous agent. Our object of study is a controlled, fixed three-attempt protocol, which lets us vary only the information returned between attempts. An agent harness changes many factors at once, including tool use, exploration and self-generated tests. Section 8 states this scope limit, and we view agent-level protocol studies as important future work. One caveat also needs stating here: both prompts include an oracle-derived line range (Section 8), so our setting is easier than standard agent evaluation.

Minor — Related work in the Introduction. Fixed. The Introduction now cites prompt and inference-setting sensitivity work (Lu et al., 2022; Shao et al., 2026) and self-repair work (Chen et al., 2024; Olausson et al., 2024).

Reviewer_evvFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how evaluation protocols affect measured LLM program-repair performance. Structured test feedback produces large improvements on the synthetic BenchCraft benchmark but no clear advantage over independent resampling on BugsInPy. The results show that benchmark scores depend substantially on the evaluation and execution protocol.

Critical strengths
  1. The paper asks an important evaluation question: benchmark results conflate model capability with the opportunities and information supplied by the evaluation procedure.
  2. The paper is well written and easy to follow.
Critical weaknesses
  1. The study covers only 30 synthetic tasks and 20 real defects. BenchCraft is also introduced by the same author, which increases the need for validation on established external benchmarks.
  2. The two experiments differ not only in benchmark realism but also in inference regime: provider-default reasoning is used in one setting and reasoning is disabled in the other. This prevents attribution of the change in protocol effect specifically to synthetic versus real-world tasks.
  3. Important reproducibility details—including full prompts, sampling parameters, feedback formatting, and patch-application rules—are not presented in the paper.
Questions for authors
  1. Could you test the model on more external datasets?
  2. Could you control the condition to show where the performance gap comes from?
  3. Please provide more details to help the reader to reproduce the results.
1 comment
AuthorsOpenReview ↗

Response to questions and concerns

We thank the reviewers for their feedback. We revised the paper as follows:

  1. New primary SWE-bench Verified experiment (Section 5, Appendix A). We ran 290 repository-stratified instances with Claude Sonnet 5 (adaptive thinking), comparing P1 and P3 at K=3 with a shared first attempt and official SWE-bench grading. P3 raises resolution from 72.8% to 79.3% (+6.6 pp, 95% CI [2.7, 10.4], exact McNemar p = 0.0013).

  2. BenchCraft has become supporting evidence (Appendix C); the main claims rely on two established real code benchmarks.

  3. BugsInPy rerun with reasoning on/off (Section 6, Tables 3–4): 3 models × 2 settings × 2 replicates. P3 > P2 > P1 in both settings; no contrast is conclusive.

  4. Failure-stage analysis (Tables 2, 5; Appendix B) identifies where P3 helps.

  5. Positioning. We discuss Chen et al. (2024) and Olausson et al. (2024) in the Introduction and Related Work and revise the contributions.

  6. Reproducibility. Appendices A–C provide full prompt templates, feedback formats, sampling/reasoning settings, patch-application rules, and statistical formulas.

Answers to Questions:

W1 / Q1 — Small scale, self-introduced synthetic benchmark; test on more external datasets. We agree, and the revision changes the evidence base accordingly. The primary experiment is now 290 SWE-bench Verified instances with official grading (Section 5). The second experiment is BugsInPy (20 defects, now 720 trials across 3 models and 2 reasoning settings). Both are established external benchmarks. BenchCraft is moved to Appendix C and is not used for any headline claim.

W2 / Q2 — Reasoning regime confounded with benchmark; control conditions to locate the gap. • BugsInPy now runs both reasoning-enabled and reasoning-disabled for every model (Tables 3–4). Enabling reasoning raises every protocol's success rate by 7.5–9.2 pp. However, the ordering P3 > P2 > P1 and the absence of a conclusive difference hold in both settings. Disabled reasoning therefore does not explain the BugsInPy result. • To locate the gap, we stratify by the stage at which the shared first attempt failed (Table 2; Table 5; Appendix B "Test-feedback subgroup"). The P3 advantage concentrates where test feedback actually exists (SWE-bench tests-failed stratum: +21.4 pp, p = 0.0015). On BugsInPy, most failures end before any test feedback is produced. • P2 (previous patch + generic message) serves as a control on BugsInPy. P3−P2 isolates feedback content from context retention: +3.3 pp, inconclusive.

W3 / Q3 — Reproducibility details. These are now in the paper: • Full prompt templates for both benchmarks (Appendices A and B). • Exact P2 and P3 feedback text for every failure stage, including the truncation rule. • Model identifiers, reasoning parameters as sent on the request path, output ceilings, and temperature and seed handling. • Patch-application rules (git apply flags, path rewriting), the stopping and acceptance rules, and FLAKY handling. • Docker image digest, BugsInPy commit hashes and Python versions. • The full list of 290 SWE-bench instances. • Closed-form statistical formulas and the bootstrap seed.

Reviewer_EyYBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how evaluation protocols affect measured LLM program-repair performance under matched maximum generation budgets. It compares independent sampling, refinement with generic feedback, and refinement with structured test feedback across three models. On 30 synthetic BenchCraft tasks, structured feedback improves visible-test success and validated repair success, although the registered precision target is not met. A separate experiment on 20 BugsInPy defects finds no clear advantage over independent sampling. The paper argues that benchmark scores should be interpreted together with the protocol and execution conditions that produced them.

Critical strengths
  1. The paper addresses a relevant evaluation problem by examining how feedback and repeated sampling change reported program-repair performance.
  2. The registered analysis, held-out validation on BenchCraft, and disclosure of inconclusive results improve the transparency of the empirical findings.
Critical weaknesses
  1. The real-world experiment offers limited insight into the conditions under which structured feedback helps. The P3−P1 difference is 4.2 percentage points, with a defect-clustered interval of [−3.3, +12.5], leaving meaningful benefits unresolved. Across Experiment B, 75.9% of terminal episodes end at patch application, before test-level feedback becomes available. However, failure-stage frequencies are not reported separately by protocol, and the feedback returned by P3 after application failures is unspecified. It is therefore difficult to assess how often P3 delivers informative feedback. The simultaneous changes in benchmark and reasoning setting further limit interpretation of the cross-experiment comparison, as the authors acknowledge. These results establish an inconclusive outcome under one configuration, but provide limited guidance on when feedback is useful.
  2. The contribution beyond existing self-repair studies needs clearer positioning and stronger analytical documentation. Olausson et al. (2024) compare self-repair with independent sampling under budget constraints and investigate feedback quality, while Chen et al. (2024) study self-debugging. Neither is discussed, leaving unclear what additional understanding the protocol-centered framing and held-out validation provide. Experiment A does offer clear directional evidence; missing the precision target does not negate that result. However, the provided manuscript does not report the sample-size justification or the design-adequacy and stopping results mentioned in Section 4. The model specification, priors, and consequences of the disclosed implementation corrections and switch to NUTS also require fuller documentation. These details are necessary to assess the strength of the registered evidence and the robustness of the reported estimates.

References for the above concerns:

  1. Olausson et al. (2024). Is Self-Repair a Silver Bullet for Code Generation? ICLR. https://arxiv.org/abs/2306.09896
  2. Chen et al. (2024). Teaching Large Language Models to Self-Debug. ICLR. https://arxiv.org/abs/2304.05128
Questions for authors
  1. What feedback does P3 receive after patch-application failures, and how do failure-stage frequencies differ across protocols?
  2. Please provide the registered sample-size justification, design-adequacy results, and sensitivity of the main contrasts to the disclosed analysis corrections.
  3. What new empirical insights does this study provide beyond prior work?
1 comment

We thank the reviewers for their feedback. We revised the paper as follows:

  1. New primary SWE-bench Verified experiment (Section 5, Appendix A). We ran 290 repository-stratified instances with Claude Sonnet 5 (adaptive thinking), comparing P1 and P3 at K=3 with a shared first attempt and official SWE-bench grading. P3 raises resolution from 72.8% to 79.3% (+6.6 pp, 95% CI [2.7, 10.4], exact McNemar p = 0.0013).

  2. BenchCraft has become supporting evidence (Appendix C); the main claims rely on two established real code benchmarks.

  3. BugsInPy rerun with reasoning on/off (Section 6, Tables 3–4): 3 models × 2 settings × 2 replicates. P3 > P2 > P1 in both settings; no contrast is conclusive.

  4. Failure-stage analysis (Tables 2, 5; Appendix B) identifies where P3 helps.

  5. Positioning. We discuss Chen et al. (2024) and Olausson et al. (2024) in the Introduction and Related Work and revise the contributions.

  6. Reproducibility. Appendices A–C provide full prompt templates, feedback formats, sampling/reasoning settings, patch-application rules, and statistical formulas.

Responses to questions:

Q1 — What feedback does P3 receive after patch-application failures, and how do failure stages differ by protocol?

After application failure, P3 receives "The provided patch could not be applied to the source file" plus the literal git apply error. After test failure it receives the pass count plus the full test stdout (BugsInPy, truncated at 4,000 chars) or the aggregate counts only (SWE-bench). After token exhaustion, it receives P2's generic message. Templates: Appendix A (SWE-bench), Appendix B (BugsInPy).

Failure stages by protocol (BugsInPy, Table 5): Patch-application failures: P3=P2 (108/240), versus P1 (99/240). Test failures: P3 (39/240), P2 (47/240), P1 (60/240). Among applied patches, test-pass rates are 70.5% (P3), 64.4% (P2), and 57.4% (P1). P3's higher success is associated with better outcomes after application, not fewer application failures.

When does P3 deliver informative feedback? With reasoning enabled, 24 BugsInPy units (11 defects) had genuine test feedback after attempt 1. In that subgroup P3 scored 6/24 and P1 4/24 (+8.3 pp, CI [−10.0, +30.4], p = 0.69). On SWE-bench, 258 (P1) and 253 (P3) of 290 instances reached test execution at least once. The P3 advantage is largest when the first patch applied but failed the tests: +21.4 pp, p = 0.0015, n = 70. After application failures it is +5.7 pp (n.s., n = 35) (Table 2).

Q2 — Sample-size justification, design adequacy, and sensitivity to the corrections.

See Appendix C.

Sample size: N=30 was compared with N=33, the entire eligible pool. Both gave identical pass/fail predictions, and N=33 narrowed intervals by only ~5%.

Design adequacy: Table 6 compares design-time predictions with realized results. The VTS precision criterion was predicted to pass (width 0.089) but failed (0.192). The VRS criterion was stated in advance to be unattainable. H2 holds for 2 of 3 model pairs. We also explain the shortfall: heterogeneity was larger than assumed (σ_p = 2.57, σ_q = 3.57).

Model, priors, sampler: we give the full two-part hierarchical specification with priors, and we disclose that an earlier circulated prior specification matched no executed stage. NUTS ran 4×1,000 draws with R̂ ≤ 1.0045, min ESS 446 and no divergences.

Sensitivity: Table 7 applies the corrections one at a time (VRS estimand, Jacobian, NUTS, model averaging). The P3−P1 VRS contrast stays between +0.21 and +0.31, and every interval excludes zero. The corrections change magnitude and width, never direction.

Q3 — Contribution beyond Olausson et al. (2024) and Chen et al. (2024).

Protocol effects on real-world defects.

Prior work investigates self-debugging on standalone code-generation tasks (Chen et al., 2024) and compares self-repair with independent sampling on HumanEval and APPS (Olausson et al., 2024). We extend these investigations to real defects in open-source projects from two established benchmarks, SWE-bench Verified and BugsInPy, where models modify existing project code and patches are evaluated against project tests. On SWE-bench Verified, structured refinement (P3) raises officially graded resolution over independent sampling (P1) by a statistically significant 6.6 percentage points (pp). On our 20-defect BugsInPy sample, the estimated advantage is similar in direction and magnitude (+5.0 pp), but statistically inconclusive (Sections 5–6).

Failure conditions associated with structured-refinement effectiveness.

We investigate how the failure stage of an initial repair attempt relates to subsequent protocol effectiveness. On SWE-bench Verified, structured refinement shows its largest observed advantage over independent sampling when the initial patch applies but fails testing, rather than failing during patch application. This suggests that executable diagnostic feedback may be an important condition for effective refinement (Section 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.