- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares six feedback types for iterative code refinement on 40 Codeforces problems across three difficulty levels. Feedback helps on easy and medium problems, but on hard problems even ground-truth hints give little gain, and best-of-N parallel sampling with the same budget scores highest. The authors conclude that refinement on hard problems acts as conditional resampling.
Critical strengths
-
The question is practical and the design is controlled: all feedback types start from the same initial program with the same budget. This isolates the effect of feedback from other agent design choices.
-
The contrast between medium and hard problems for ground-truth hints (40.4% versus 5.3% gain) is an informative observation. The trend is broadly consistent across five models.
Critical weaknesses
-
The refinement is memoryless. Each round sees only the previous program and its feedback. The finding that trajectories fluctuate and resemble resampling may therefore follow from this design, since practical agents keep the full history.
-
The scale is small and the setup may affect the hard-problem results. For example, 40 problems with only 10 hard ones, selected by test size, solved in Python with relaxed limits and scored by the fraction of tests passed. The correlation in Figure 5 is also partly mechanical, because both axes contain the initial score.
Questions for authors
-
What happens when the agent sees the full history of previous attempts and feedback, or keeps the best program so far?
-
For ground-truth hints on hard problems, are the failures due to wrong algorithms or to implementation and efficiency issues in Python? Reporting the full-solve rate in addition to the pass rate would help.
We thank the reviewer for their feedback. Here are our responses to the raised concerns:
Weaknesses:
Following your suggestion, we ran a new experiment. In this experiment, we kept the full history of the model’s previous generations and feedback, only dropping earlier conversations when the context was nearly full. The results are as follows:
Using a limited number of questions was intentional; we initially picked only questions that could provide at least 10 test cases that fit within the model’s context. We designed another experiment where, instead of passing raw test cases in the context, we just provide the head and the tail of arrays within the context, in addition to the differences between expected outputs and generated outputs. We therefore doubled the number of questions in our experiments:
These experiments show that our main claims are not artifacts of the experiment setup; more concretely, Parallel Best-of-10 outperforms other feedback types on hard problems, while Ground Truth Hints are most effective on easier problems.
Regarding the informativeness of Figure 5, we added two new plots to the updated document: 1) Figure 15 shows refinement gain vs the initial performance. This figure shows that in a majority of trajectories where the initial program performs <20%, the refinement gain is still very limited. 2) Figure 16 shows best-of-parallel vs the performance of the refined program. This figure shows that in the majority of cases, the final program is below best-of-parallel even when best-of-parallel is not 100%. Both figures validate our claim that the refinement gain diminishes as the initial gap narrows.
Questions
Please see our response to the first weakness.
While evaluating the generated responses, we made sure that the model does not fail due to efficiency issues in Python. In particular, for each problem in our inventory, we used a human-written reference code to ensure that the problem is solvable.
We additionally report the full-solve rate in the following table, as you suggested. In the following table, a question is considered solved only if it passes all the test cases and we report the average solve rate across all questions.
Thank you again for your reviews and please let us know if you have further concerns.