Repair Rate Is Not Repair: A Counterfactual Audit of Cross-Model Code Critique

Critique-and-revise pipelines are routinely credited with the improvements that follow them: a critic comments on a program, the program is revised, it passes its tests, and the critic is scored as having repaired it. That is a causal claim, almost never tested against the obvious control: what the same reviser would have produced from the same program given no substantive advice. We run that control. In an existing three-round experiment on HumanEval+ whose round-0 programs are shared byte-identically across conditions, we form matched pairs at the only transition where the starting state is fixed and compare each advice condition against a null-advice arm whose critique is the fixed string NO ISSUES FOUND. For a neutral cross-model critic, the naive repair rate reports 0.034 while the counterfactual uplift is -0.041 (95% CI [-0.061, -0.022]); with an added "senior engineer" persona the figures are 0.011 and -0.037 (95% CI [-0.055, -0.018]). The naive statistic is computed only over initially failing programs. On that stratum the causal uplift is +0.011 for the neutral critic and -0.011 for the persona variant, neither resolvable away from zero, so neither is what drives the inversion. What the naive rate excludes by construction is the already-passing stratum, where essentially all of the harm falls. A pipeline scored by repair rate therefore records a benefit while the full matched comparison records a harm. The mechanism is visible in the data: the cross-model critic reports a defect in 94.5% of code that already passes its tests, and breaks such programs at 5.2% against a 0.2% null-advice baseline. Harmful-to-helpful flip ratios are 21:1 for the neutral critic and 10:1 for the persona variant, 41:3 pooled. These are measurements of one model pair on one benchmark, and our self-critique arm is degenerate and supports no conclusion about self-critique.