acceptodds
Under review as a conference paper at ICLR 2027

Repair Rate Is Not Repair: A Counterfactual Audit of Cross-Model Code Critique

Abstract

Critique-and-revise pipelines are routinely credited with the improvements that follow them: a critic comments on a program, the program is revised, it passes its tests, and the critic is scored as having repaired it. That is a causal claim, almost never tested against the obvious control: what the same reviser would have produced from the same program given no substantive advice. We run that control. In an existing three-round experiment on HumanEval+ whose round-0 programs are shared byte-identically across conditions, we form matched pairs at the only transition where the starting state is fixed and compare each advice condition against a null-advice arm whose critique is the fixed string NO ISSUES FOUND. For a neutral cross-model critic, the naive repair rate reports 0.034 while the counterfactual uplift is -0.041 (95% CI [-0.061, -0.022]); with an added "senior engineer" persona the figures are 0.011 and -0.037 (95% CI [-0.055, -0.018]). The naive statistic is computed only over initially failing programs. On that stratum the causal uplift is +0.011 for the neutral critic and -0.011 for the persona variant, neither resolvable away from zero, so neither is what drives the inversion. What the naive rate excludes by construction is the already-passing stratum, where essentially all of the harm falls. A pipeline scored by repair rate therefore records a benefit while the full matched comparison records a harm. The mechanism is visible in the data: the cross-model critic reports a defect in 94.5% of code that already passes its tests, and breaks such programs at 5.2% against a 0.2% null-advice baseline. Harmful-to-helpful flip ratios are 21:1 for the neutral critic and 10:1 for the persona variant, 41:3 pooled. These are measurements of one model pair on one benchmark, and our self-critique arm is degenerate and supports no conclusion about self-critique.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, code generation, self-refinement, LLM critique, causal inference, counterfactual evaluation, evaluation methodology, HumanEval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_msPTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper questions the practice of directly treating “how many buggy programs were fixed after accepting critique” as the critic’s contribution. Under the same initial code and the same revision model, the authors compare real critique with a fixed NO ISSUES FOUND control and decompose the final outcomes into beneficial, no-difference, and harmful paired results. The main finding is: the naive fix rate of neutral cross-model critique is 3.4%, but its overall success rate difference relative to the control is −4.1 percentage points, with the harm mainly occurring on programs that had already passed the tests.

Critical strengths

None

Critical weaknesses
  1. The main structure of the paper is missing, and the overall quality does not meet the ICLR standard: the paper does not even include a Method section.
  2. The paper's writing logic is very poor and difficult to follow.
Questions for authors

Seed Weeknesses.

Reviewer_JU3GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits whether improvements after code critique can be attributed to the critic. It reanalyzes a HumanEval+ experiment using Qwen2.5-Coder-7B-Instruct as generator/reviser and DeepSeek-Coder-6.7B-Instruct as cross-model critic, with both models quantized to 4 bits. Five conditions share byte-identical initial programs across 164 problems and three seeds. The neutral and persona-based cross-model conditions have estimated uplifts of −4.1 and −3.7 percentage points, respectively. Almost all observed harm occurs on initially passing programs; effects on initially failing programs are small and uncertain. The paper also inspects harmful revisions and explicitly excludes substantive conclusions about self-critique because that arm always emits the clean-code verdict. Its proposed evaluation practice is to include a revision control and report both repair and regression.

Critical strengths
  • The supplementary checkpoints and analysis scripts reproduce the 9,840 trajectory rows, 1,476 paired outcomes, and main effect estimates, providing a concrete basis for auditing the quantitative findings.
Critical weaknesses
  • The paper lacks established repair baselines and tests only one model pair on HumanEval+ (Sections 2–3). The result could therefore reflect a poor critic or this benchmark's tasks, leaving little evidence that the paper identifies a problem affecting established repair methods. Please evaluate established repair methods and test whether the finding holds on another independent coding benchmark.
  • The paper's main evidence of harm comes from changing code that already works, while the effect on broken code is inconclusive (Table 3). A system that repairs only failed programs would skip the working code responsible for most of this harm, so the headline result gives little guidance about whether critique helps that system. Please compare revision with and without critique in a system that tests code first and repairs only failures.

The released data and scripts make the reported results easy to check. However, the missing established baselines and reliance on HumanEval+ leave the evaluation too narrow to support a rigorous assessment of critique-based repair. Its practical relevance is also unclear because the main harm concerns working code and the effect on broken code remains unresolved. I therefore lean toward weak rejection.

Questions for authors
  1. What results from established critique-and-repair methods on an independent coding benchmark support your conclusion that critique causes more harm than benefit?
  2. When only programs that fail their tests are revised, how does the success rate with critique compare with revision alone?
Reviewer_Ph14OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper reanalyzes a critique-and-revise experiment on HumanEval+. It compares critique-guided revisions with revisions receiving fixed “NO ISSUES FOUND” feedback. In the tested setting, cross-model critique reduces overall test performance, mainly by damaging initially passing programs. The authors argue that repair rates alone do not establish the critic’s contribution.

Critical strengths

The paper compares several conditions, including self-critique, cross-model critique, a persona variant, and a fixed-feedback control. The analysis of the experiment is informative.

Critical weaknesses
  1. The motivation relies mainly on critique-and-revise studies from 2023–2024. Current coding agents can inspect, modify, and test code within an integrated workflow, without a separately designated critic and reviser. The paper does not explain how its setting relates to these systems or what useful guidance its findings provide. I therefore find its practical significance and contribution to the community limited.

  2. The experimental setup is poorly justified. Test results are available, but neither model receives them. Even passing programs enter revision, with no clear explanation of what practical setting this represents. The paper also lacks a systematic evaluation of whether critiques identify real bugs. This leaves incorrect feedback and unsuccessful revision insufficiently separated. Using only one pair of older 6.7B–7B models further limits what can be learned from the results.

  3. The manuscript provides prompts and implementation details, but I could not locate access links to the stored trajectories, pipeline notebooks, or analysis scripts. These materials are particularly important because the study depends entirely on previously collected outputs. This limits my confidence in the reported results.

Questions for authors
  1. Why are test outcomes withheld from both models? What intended use case motivates revising all programs instead of only those that fail the tests?

  2. Among the 87 initially failing instances, how often did the critic identify the actual bug? When the diagnosis was correct, how often did the reviser successfully implement the correction?

  3. Why is “NO ISSUES FOUND” an appropriate control for programs that actually fail? Have you compared it with empty feedback or a neutral request to inspect and revise the code?

  4. Why are the main repair rates and causal uplifts reported as pooled results rather than per-seed estimates with a mean and standard deviation? Are the findings consistent across all three seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.