acceptodds
Under review as a conference paper at ICLR 2027

When Does Feedback Help? Evaluating Feedback for Iterative Refinement in Coding Agents

Abstract

Recent advances in AI increasingly rely on agents that iteratively refine their responses at test time using feedback from an environment. Yet it remains unclear what type of feedback most effectively enables an agent to make discoveries beyond its existing capabilities. We investigate this question using a controlled testbed of coding problems spanning different difficulty levels. We compare six forms of feedback most commonly used in practice: detailed execution reports, aggregate performance scores, self-reflection, external LLM code review, failures on LLM-generated tests, and ground-truth hints written by human experts. Using this framework, we show that feedback effectiveness depends strongly on problem difficulty: On easy problems, nearly all feedback types enable the agent to reach full accuracy when the agent was unable to solve them in the initial attempt. On medium problems, feedback derived from ground-truth solutions yields the largest improvement (40.4%), followed by feedback from LLM-generated test cases (24.1%). On hard problems, however, even ground-truth hints provide only a marginal improvement (5.3%). In this case, feedback based on LLM-generated test cases remains the most effective (20.4%). Nevertheless, under the same inference budget, best-of-N parallel sampling outperforms every iterative-feedback method in hard problems. In fact, we show that the gains from refinement are proportional to the gap between the initial response and the best solution obtainable through parallel sampling. These findings reveal an illusion of sequential refinement: on difficult problems, iterative refinement primarily helps the agent recover from poor initial generations and approach, but not surpass, the best solutions it could independently generate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding Agents, Agentic Harness, AI for Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aGYTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares six feedback types for iterative code refinement on 40 Codeforces problems across three difficulty levels. Feedback helps on easy and medium problems, but on hard problems even ground-truth hints give little gain, and best-of-N parallel sampling with the same budget scores highest. The authors conclude that refinement on hard problems acts as conditional resampling.

Critical strengths
  1. The question is practical and the design is controlled: all feedback types start from the same initial program with the same budget. This isolates the effect of feedback from other agent design choices.

  2. The contrast between medium and hard problems for ground-truth hints (40.4% versus 5.3% gain) is an informative observation. The trend is broadly consistent across five models.

Critical weaknesses
  1. The refinement is memoryless. Each round sees only the previous program and its feedback. The finding that trajectories fluctuate and resemble resampling may therefore follow from this design, since practical agents keep the full history.

  2. The scale is small and the setup may affect the hard-problem results. For example, 40 problems with only 10 hard ones, selected by test size, solved in Python with relaxed limits and scored by the fraction of tests passed. The correlation in Figure 5 is also partly mechanical, because both axes contain the initial score.

Questions for authors
  1. What happens when the agent sees the full history of previous attempts and feedback, or keeps the best program so far?

  2. For ground-truth hints on hard problems, are the failures due to wrong algorithms or to implementation and efficiency issues in Python? Reporting the full-solve rate in addition to the pass rate would help.

1 comment

We thank the reviewer for their feedback. Here are our responses to the raised concerns:

Weaknesses:

  1. Following your suggestion, we ran a new experiment. In this experiment, we kept the full history of the model’s previous generations and feedback, only dropping earlier conversations when the context was nearly full. The results are as follows:

    configEasyMediumHard
    Initial90.953.034.2
    Detailed Execution Report96.5 ± 1.268.4 ± 3.539.2 ± 3.9
    LLM-Generated Test96.2 ± 1.268.2 ± 3.450.7 ± 3.5
    Ground Truth Hints98.4 ± 0.890.3 ± 1.934.9 ± 3.5
    Parallel Best-of-1095.884.055.5
  2. Using a limited number of questions was intentional; we initially picked only questions that could provide at least 10 test cases that fit within the model’s context. We designed another experiment where, instead of passing raw test cases in the context, we just provide the head and the tail of arrays within the context, in addition to the differences between expected outputs and generated outputs. We therefore doubled the number of questions in our experiments:

    configEasyMediumHard
    Initial87.246.826.3
    Detailed Execution Report95.6 ± 0.969.4 ± 2.031.5 ± 2.2
    LLM-Generated Test91.5 ± 1.372.1 ± 1.835.9 ± 2.1
    Ground Truth Hints98.9 ± 0.493.2 ± 1.134.3 ± 2.3
    Parallel Best-of-1098.174.039.6

    These experiments show that our main claims are not artifacts of the experiment setup; more concretely, Parallel Best-of-10 outperforms other feedback types on hard problems, while Ground Truth Hints are most effective on easier problems.

    Regarding the informativeness of Figure 5, we added two new plots to the updated document: 1) Figure 15 shows refinement gain vs the initial performance. This figure shows that in a majority of trajectories where the initial program performs <20%, the refinement gain is still very limited. 2) Figure 16 shows best-of-parallel vs the performance of the refined program. This figure shows that in the majority of cases, the final program is below best-of-parallel even when best-of-parallel is not 100%. Both figures validate our claim that the refinement gain diminishes as the initial gap narrows.

Questions

  1. Please see our response to the first weakness.

  2. While evaluating the generated responses, we made sure that the model does not fail due to efficiency issues in Python. In particular, for each problem in our inventory, we used a human-written reference code to ensure that the problem is solvable.

    We additionally report the full-solve rate in the following table, as you suggested. In the following table, a question is considered solved only if it passes all the test cases and we report the average solve rate across all questions.

    configEasyMediumHard
    Initial79.333.68.0
    Detailed Execution Report92.7 ± 2.164.3 ± 4.025.0 ± 4.3
    LLM-Generated Test87.3 ± 2.765.7 ± 4.012.0 ± 3.2
    Ground Truth Hints93.3 ± 2.085.7 ± 3.016.0 ± 3.7
    Parallel Best-of-1086.757.120.0

Thank you again for your reviews and please let us know if you have further concerns.

Reviewer_6PVdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates which types of feedback are most effective for iterative refinement in coding agents and how their effectiveness varies with problem difficulty. The authors evaluate six different feedback types on 40 coding problems collected from Codeforces, comparing their effectiveness based on test-case pass rates. The results show that ground-truth hints are most effective on medium-difficulty problems, while LLM-generated test feedback performs best among the feedback methods on hard problems. Furthermore, the authors show that best-of-N parallel sampling outperforms iterative refinement on hard problems under the same generation budget. Based on these findings, they argue that iterative refinement primarily helps models recover solutions already within their capabilities rather than discover fundamentally new solutions.

Critical strengths

None

Critical weaknesses

1. Insufficient evidence for the central claim and inconsistent interpretation of results.

Best-of-N relies on oracle-based selection using evaluation test results, which cannot be regarded as a well-defined boundary of a model's existing capabilities. Therefore, outperforming or failing to outperform this baseline does not establish whether iterative refinement enables a model to go beyond its existing capabilities. Furthermore, Figure 14 in the appendix shows that refinement with detailed execution feedback outperforms best-of-N under best-intermediate selection, contrary to the authors' stated conclusion. Although this evaluation differs from the main experiment, the discrepancy is not adequately addressed, weakening the paper's central claim.

2. Limited generalizability and insufficient analysis of feedback effectiveness.

The experiments rely on a small pool of only 40 problems from a single source (Codeforces), including just 10 hard problems. This makes it difficult to determine whether the observed differences among feedback strategies generalize to other coding tasks or are specific to the selected problems. Moreover, the six feedback conditions cover only a limited set of possible feedback mechanisms, and their effectiveness may depend on prompt design, information quality, and feedback granularity. Without systematically examining these factors, it remains unclear whether the observed limitations arise from iterative refinement itself or from the particular feedback configurations evaluated.

Questions for authors
  • What evidence supports treating oracle-selected best-of-N performance as a boundary of a model's existing capabilities?
  • How do you reconcile the central claim that best-of-N outperforms iterative refinement on hard problems with Figure 14, where Detailed Execution Report achieves 60.8% compared to 58.3% for best-of-N? Is this difference statistically significant?
  • Given that the experiments use only 40 problems from Codeforces, including 10 hard problems, what evidence supports the generalizability of the findings beyond this particular problem set? Are the observed trends robust to different problem selections or difficulty classifications?
  • How sensitive are the results to feedback prompt design, information quality, and granularity? Is there evidence that the observed limitations of iterative refinement persist under alternative feedback configurations?
1 comment

We thank the reviewer for their feedback. Here are our responses to the raised concerns:

Questions

  1. We agree with the reviewer that best-of-parallel is not necessarily a well-defined boundary of a model's existing capabilities. However, it is the most common metric used for comparing and evaluating models in the context of test-time scaling. We are happy to reword our claims if that would address the reviewer’s concern.

  2. Thanks for catching the inconsistency between Figure 14 and the appendix text. The difference is statistically significant, but the detailed execution feedback is still not substantially better than parallel sampling, as the gap is not large.

  3. Using a limited number of questions was intentional; we initially picked only questions that could provide at least 10 test cases that fit within the model’s context. We designed another experiment where, instead of passing raw test cases in the context, we just provide the head and the tail of arrays within the context, in addition to the differences between expected outputs and generated outputs. We therefore doubled the number of questions in our experiments:

    configEasyMediumHard
    Initial87.246.826.3
    Detailed Execution Report95.6 ± 0.969.4 ± 2.031.5 ± 2.2
    LLM-Generated Test91.5 ± 1.372.1 ± 1.835.9 ± 2.1
    Ground Truth Hints98.9 ± 0.493.2 ± 1.134.3 ± 2.3
    Parallel Best-of-1098.174.039.6

    The experiments above show that our main claims are not artifacts of the experiment setup; more concretely, Parallel Best-of-10 outperforms other feedback types on hard problems, while Ground Truth Hints are most effective on easier problems.

  4. We used standard prompting the exact prompting that the benchmark used. Regarding the feedback prompting, we used minimal textual description (please see Appendix A for the exact prompts). Given that modern LLMs are robust to small variations in prompting, we expect the results to generalize to other prompts as well.

    Regarding the quality of feedback, we evaluated different types of feedback as the main focus of our paper. In the case of ground-truth feedback, we directly used solutions that were authored by human editors. We agree that controlling the amount of information in the feedback is a valuable direction, but it requires inventing new tools to control the granularity and amount of information, which would not fit within the scope of our work.

Thank you again for your reviews and please let us know if you have further concerns.

Reviewer_xv7EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper examines which forms of feedback allow a coding agent to improve on its initial attempt. Using 40 LiveCodeBench-Pro problems at three difficulty levels, the authors compare six feedback signals: detailed execution reports, aggregate pass rates, self-reflection, critiques from a separate LLM reviewer, failures on LLM-generated tests, and hints drawn from human-written editorials. Every refinement loop starts from the same initial program. The main experiments use gpt-oss-120b, and results for four further models are reported in the appendix. Nearly every signal suffices on easy problems and editorial hints help most on medium problems, whereas on hard problems hints add only about five points and LLM-generated tests become the most effective signal. Under a matched call budget, best-of-N sampling outperforms all feedback types on hard problems, which leads the authors to characterize sequential refinement as conditional resampling that cannot move beyond the frontier reached by parallel sampling.

Critical strengths
  1. The experimental control is the main asset of the paper. Because every condition shares the starting program and decoding configuration, differences in outcome can be attributed to the feedback signal itself, which is impossible to disentangle in existing agent frameworks that vary many components at once. The observation that LLM-generated tests are at least as informative as the official test cases on hard problems, 56.2 versus 49.1 in Figure 3, is unexpected and has clear practical value, since ground-truth tests are seldom available outside benchmarks.
  2. The finding that editorial hints barely help once a problem exceeds the model's ability is a useful caution for anyone building discovery-oriented agents around feedback loops.
Critical weaknesses
  1. The central claim rests on an asymmetric comparison that the appendix itself undermines. Best-of-N is credited with the highest-scoring sample on the evaluation tests, which amounts to oracle selection, whereas refinement is judged only by its final program. When Appendix B.2 applies the same best-so-far selection to refinement, the detailed execution report reaches 60.8 and LLM-generated tests 58.6 on hard problems, against 58.3 for best-of-N. Refinement is therefore on par with, or slightly ahead of, parallel sampling. The accompanying text nonetheless states that best-of-N continues to outperform refinement, and it names the detailed execution report as the strongest signal on hard problems, contradicting the main text, which gives that role to LLM-generated tests. A fair reading of the evidence is that refinement fails to retain its best program, a considerably narrower conclusion than an illusion of refinement. I would also question how much Figure 5 adds, because a negative relation between the initial gap and the eventual gain follows almost automatically from the fact that weaker starting programs leave more room for improvement.
  2. The refinement loop conditions only on the previous program and its feedback, with no access to earlier attempts. Under that design, the behavior in Figure 7, where revisions discard progress and reintroduce old errors, is to be expected rather than revealing. Practical systems such as SWE-agent and Reflexion retain the interaction history or at least the best attempt so far. The hard split also contains only ten problems, the error bars treat repeated runs on the same problem as independent observations, and several of the hard-problem differences are around two points. At this scale, the statement in the introduction that agents cannot make novel advances through interaction with the environment is not supported.
Questions for authors
  1. How do the authors reconcile Figure 14 with the headline claim? If refinement under the same selection rule matches best-of-N on hard problems, should the conclusion not be restated as a failure to retain progress? My assessment depends heavily on this point.
  2. Was a variant with access to the full history, or to the best program found so far, evaluated? If the fluctuations in Figure 7 disappear under such a variant, the conditional-resampling interpretation would need to be revised.
  3. What correlation does Figure 5 show when each refinement step is replaced by an independent fresh sample? A comparable value would indicate that the figure reflects regression to the mean rather than the proposed mechanism.
  4. What fraction of the LLM-generated tests carry incorrect expected outputs on hard problems, and does their advantage survive once test length is controlled, given that the official tests are far longer?
1 comment

We thank the reviewer for their feedback. Here are our responses to the raised concerns:

Critical Weaknesses

  1. Best-of-N selection can indeed improve the performance of the sequential refinement, but it cannot outperform parallel sampling in the majority of cases (except the 60.8 vs 58.3 case, where the difference is not large). This reinforces the statement that the improvement in sequential refinement is mostly due to retrying sampling rather than to meaningful progress.

    As you correctly noted, part of this failure is not being able to retain the past progress and jumping to an entirely new solution. This is a central message we conveyed in Section 4.3. We emphasize that best-of-N combined with sequential refinement uses more privileged knowledge than best-of-N Parallel sampling, since the LLM directly sees the test cases and is therefore expected to outperform parallel sampling under similar conditions.

    Regarding the informativeness of Figure 5, we added two new plots to the updated document: 1) Figure 15 shows refinement gain vs the initial performance. This figure shows that in a majority of trajectories where the initial program performs <20%, the refinement gain is still very limited. 2) Figure 16 shows best-of-parallel vs the performance of the refined program. This figure shows that in the majority of cases, the final program is below best-of-parallel even when best-of-parallel is not 100%. Both figures validate our claim that the refinement gain diminishes as the initial gap narrows.

  2. Following your suggestions, we ran complementary experiments, including those conditioning on the entire history, conditioning on the best sequential response at every step, and extending to a larger number of questions. All of the experiments show that our main claims are not artifacts of the experiment setup; more concretely, Parallel Best-of-10 outperforms other feedback types on hard problems, while Ground Truth Hints are most effective on easier problems.

    Conditioning on the entire history: In this experiment, we kept the full history of the model’s previous generations and feedback, only dropping earlier conversations when the context was nearly full. The results are as follows:

    configEasyMediumHard
    Initial90.953.034.2
    Detailed Execution Report96.5 ± 1.268.4 ± 3.539.2 ± 3.9
    LLM-Generated Test96.2 ± 1.268.2 ± 3.450.7 ± 3.5
    Ground Truth Hints98.4 ± 0.890.3 ± 1.934.9 ± 3.5
    Parallel Best-of-1095.884.055.5

    Conditioning on the best so far: In this experiment, we sampled only from the best program so far at each iterative step, preventing the model from degrading throughout the process.

    configEasyMediumHard
    Initial90.953.034.4
    Detailed Execution Report98.8 ± 0.681.6 ± 2.748.0 ± 4.0
    LLM-Generated Test96.2 ± 1.479.0 ± 2.852.0 ± 3.2
    Ground Truth Hints99.3 ± 0.395.6 ± 1.346.7 ± 3.8
    Parallel Best-of-1095.884.055.5

    Experiments with a larger number of questions: Using a limited number of questions was intentional; we initially picked only questions that could provide at least 10 test cases that fit within the model’s context. We designed another experiment where, instead of passing raw test cases in the context, we just provide the head and the tail of arrays within the context, in addition to the differences between expected outputs and generated outputs. We therefore doubled the number of questions in our experiments:

    configEasyMediumHard
    Initial87.246.826.3
    Detailed Execution Report95.6 ± 0.969.4 ± 2.031.5 ± 2.2
    LLM-Generated Test91.5 ± 1.372.1 ± 1.835.9 ± 2.1
    Ground Truth Hints98.9 ± 0.493.2 ± 1.134.3 ± 2.3
    Parallel Best-of-1098.174.039.6

Questions:

  1. Please see our response to the first weakness.

  2. Please see our response to the second weakness.

  3. Please see our response to the first weakness; we provided Figures 15 and 16.

  4. The following table shows the performance of test cases generated by gpt-oss-120B when compared against the ground truth code. As shown in the table, the majority of test cases generated by the model are correct, even in hard questions.

    difficultytestspass rate
    easy30093.0%
    medium28094.3%
    hard22097.7%
    all80094.8%

Thank you again for your reviews, and please let us know if you have further concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.