- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper questions the practice of directly treating “how many buggy programs were fixed after accepting critique” as the critic’s contribution. Under the same initial code and the same revision model, the authors compare real critique with a fixed NO ISSUES FOUND control and decompose the final outcomes into beneficial, no-difference, and harmful paired results. The main finding is: the naive fix rate of neutral cross-model critique is 3.4%, but its overall success rate difference relative to the control is −4.1 percentage points, with the harm mainly occurring on programs that had already passed the tests.
Critical strengths
None
Critical weaknesses
- The main structure of the paper is missing, and the overall quality does not meet the ICLR standard: the paper does not even include a Method section.
- The paper's writing logic is very poor and difficult to follow.
Questions for authors
Seed Weeknesses.