acceptodds
Under review as a conference paper at ICLR 2027

Action, Not Belief: What Makes Hypothesis Prompting Work in Debugger Agents

Abstract

Recent work has introduced hypothesis-driven debugging into LLM-based program repair, prompting agents to formulate hypotheses, predict observations, and test them with debugging tools. However, these methods combine hypothesis generation with guidance for subsequent actions, making it unclear which component drives the repair gain. We disentangle these components through controlled comparisons under a fixed debugger interface. To examine whether prediction–observation agreement tracks repair performance, we introduce HypoCal, a measure of agreement between stated runtime predictions and subsequent debugger observations. Our experiments show that injecting hypotheses into the next action prompt improves repair, whereas eliciting hypotheses alone does not. The gain disappears when the injected hypothesis concerns another bug or misidentifies the fault location. HypoCal remains low across prompting conditions and does not track repair gains. An analysis of tool use shows that hypothesis injection shifts tool use from source inspection toward runtime probing. Across four models, repair effects have the same rank ordering as baseline structured-probing rates, while injection redirects the models toward different tools. Together, these findings characterize hypothesis prompting as action shaping: its repair effect varies with the model's existing tool-use behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, automated debugging, hypothesis prompting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GCSDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focus on repairing code with agents. It finds that injecting a correctly localized hypothesis improves LLM bug repair not by making predictions more accurate, but by shifting tool use toward runtime probing, with effects varying by model's baseline debugging behavior.

Critical strengths
  1. The falsification ladder isolates action-time injection of a correctly localized hypothesis as the true cause of repair gains.
  2. It reveals gains come from action shaping, not better beliefs, and introduces the cross-model action gap.
Critical weaknesses
  1. The Inject-Test protocol is basically circular,because forcing the model to predict runtime states before acting naturally pushes it to verify those predictions later, so the "action shift" is a built-in consequence of the prompt structure, not an independent causal finding. And the HypoCal metric used to rule out "better beliefs" stays stuck at 2 to 4 percent, a severe floor effect. Drawing conclusions about prediction accuracy from such an insensitive measure is statistically weak and cannot rule out that internal understanding improved but went undetected, leaving the core "action, not belief" claim without solid causal support.
  2. The paper’s most striking claim, a perfect correlation (Spearman ρ = +1.0) between repair effects and baseline probing rates, rests on only 4 models. With n=4, any consistent ordering yields +1.0, so this is a coincidence rather than evidence. The cross-model comparison also uses only 43 instances, giving very wide confidence intervals where effects like Claude’s -7.0pp and Qwen’s +4.7pp may not even be significant. Worse, the four models span a huge baseline range (9.3% to 69.8%), creating ceiling effects, yet baseline strength is never controlled. The causal attribution to probing rate is therefore impure, and the conclusion may just be a chance ordering with no generalizability.
Questions for authors

Q1: The authors provide the effect tracks probing rate, not baseline strength. However, the models range from 9.3% to 69.8% baseline. Could the authors explain it?

Q2: The authors use Elicit to show the extra LLM call isn't enough. However Inject-Test also changes the output format. How do authors confirm that it's the hypothesis content, not just the structural change?

Reviewer_HbvVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the mechanisms underlying hypothesis-driven debugging in LLM-based software repair agents equipped with external debugging tools. It introduces a controlled experimental framework that separates hypothesis elicitation from action-time injection and examines the effects of different hypothesis formulations and content. The authors also propose HypoCal to measure prediction–observation agreement and analyze how hypothesis prompting influences agents' tool-use behavior. The study aims to understand whether the benefits of hypothesis prompting arise from improved runtime predictions or changes in action selection.

Critical strengths

S1. Important Problem of Hypothesis-Driven Debugging with External Tools

The paper investigates an important problem of understanding how hypothesis-driven debugging influences LLM-based software repair agents that interact with external debugging tools.

Critical weaknesses

W1. Limited Motivation and Significance of the Experimental Decomposition

The paper does not sufficiently motivate why disentangling hypothesis elicitation from action-time injection is an important research problem or what practical insights this distinction provides for designing debugger agents. In particular, since the elicited hypothesis in M1 is not passed to the action-selection model or otherwise used, its lack of performance improvement is largely expected by construction (c.f. “In the main Qwen evaluation, eliciting a hypothesis without including it in the next action prompt yields no significant repair gain” at Line 73). This makes one of the paper's central findings somewhat trivial and raises concerns about the significance of the proposed experimental decomposition.

W2. Insufficient Methodological Details to Assess Experimental Validity

The paper lacks a clear, self-contained description of the underlying hypothesis-driven debugging protocol and its experimental interventions. While Appendix B provides some details about M0/M1/M2, essential information about M3, M2T, M2TS, and M2W remains insufficient, particularly the exact prompt modifications and the procedures for constructing off-instance or incorrectly localized hypotheses. Without these details, it is difficult to assess whether the comparisons properly isolate the intended factors, raising concerns about the validity of the experimental findings and conclusions.

Questions for authors

Q1. Motivation and Significance of the Experimental Decomposition

The paper aims to disentangle hypothesis elicitation from action-time injection, but the practical significance of this distinction remains unclear. What nontrivial insights does this comparison provide, and how can the findings inform the design of more effective debugger agents?

Q2. Methodological Details and Validity of Experimental Controls

Could the authors provide a clearer description of the underlying hypothesis-driven debugging protocol and the exact implementation of all experimental variants, particularly M3, M2T, M2TS, and M2W? How are off-instance and incorrectly localized hypotheses constructed, and how is it ensured that each comparison isolates the intended factor? These details are essential for assessing the validity of the experimental conclusions.

Reviewer_xFHaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates why hypothesis prompting can improve the program repair performance of LLM-based debugging agents. Under a fixed tool interface and interaction budget, the authors conduct controlled experiments to isolate the effects of hypothesis generation, hypothesis injection before action selection, test-versus-falsify instructions, and hypothesis format and content. They also introduce HypoCal to measure agreement between runtime predictions and subsequent tool observations, alongside an analysis of tool-use behavior. Experiments with Qwen across five random seeds on the 300 SWE-bench Lite instances show that hypothesis injection improves the repair rate by approximately 4 percentage points on average.

Critical strengths
  1. The ablation study is relatively comprehensive, comparing hypothesis elicitation alone, hypothesis injection, free-text hypotheses, irrelevant hypotheses, and hypotheses that incorrectly localize the fault.

  2. The method adds a hypothesis-generation step at each turn and passes its output to subsequent action selection, without requiring model retraining. Overall, the approach is simple yet effective in the main experimental setting.

Critical weaknesses
  1. After hypothesis injection, both tool-use patterns and repair performance change. However, this does not establish that the changes in tool use cause the performance improvement. An alternative explanation is that the hypothesis helps the model understand the problem more accurately, which then leads to more appropriate actions. The current experiments do not distinguish between these explanations.

  2. The cross-model evaluation includes only 43 instances, with each instance accounting for approximately 2.3 percentage points. The results are therefore sensitive to the outcomes of a small number of instances. Larger-scale evaluation is needed to establish which models benefit from the intervention and which are adversely affected.

  3. Although the paper studies hypothesis generation and tool-based verification, the hypothesis output itself includes an action recommendation. It therefore remains unclear what distinct value hypothesis prompting offers over general debugging planning or action guidance.

  4. The experimental comparisons primarily consist of ablations of the proposed method, without comparisons against alternatives such as general debugging plans, action-only recommendations, or direct guidance on tool use. Consequently, while the experiments demonstrate an improvement over the selected reactive agent baseline, they do not sufficiently establish the method’s effectiveness relative to these alternatives.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.