- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focus on repairing code with agents. It finds that injecting a correctly localized hypothesis improves LLM bug repair not by making predictions more accurate, but by shifting tool use toward runtime probing, with effects varying by model's baseline debugging behavior.
Critical strengths
- The falsification ladder isolates action-time injection of a correctly localized hypothesis as the true cause of repair gains.
- It reveals gains come from action shaping, not better beliefs, and introduces the cross-model action gap.
Critical weaknesses
- The Inject-Test protocol is basically circular,because forcing the model to predict runtime states before acting naturally pushes it to verify those predictions later, so the "action shift" is a built-in consequence of the prompt structure, not an independent causal finding. And the HypoCal metric used to rule out "better beliefs" stays stuck at 2 to 4 percent, a severe floor effect. Drawing conclusions about prediction accuracy from such an insensitive measure is statistically weak and cannot rule out that internal understanding improved but went undetected, leaving the core "action, not belief" claim without solid causal support.
- The paper’s most striking claim, a perfect correlation (Spearman ρ = +1.0) between repair effects and baseline probing rates, rests on only 4 models. With n=4, any consistent ordering yields +1.0, so this is a coincidence rather than evidence. The cross-model comparison also uses only 43 instances, giving very wide confidence intervals where effects like Claude’s -7.0pp and Qwen’s +4.7pp may not even be significant. Worse, the four models span a huge baseline range (9.3% to 69.8%), creating ceiling effects, yet baseline strength is never controlled. The causal attribution to probing rate is therefore impure, and the conclusion may just be a chance ordering with no generalizability.
Questions for authors
Q1: The authors provide the effect tracks probing rate, not baseline strength. However, the models range from 9.3% to 69.8% baseline. Could the authors explain it?
Q2: The authors use Elicit to show the extra LLM call isn't enough. However Inject-Test also changes the output format. How do authors confirm that it's the hypothesis content, not just the structural change?