- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces Causal Agent Bench (CAB) to study when interventions on tool-using agents support causal conclusions about why the agents fail. Its thesis is that an intervention which improves task performance does not thereby identify the mechanism of the original failure. Using paired repository-based retrieval tasks under CLEAN, PRESERVE and DISRUPT, matched BASE/PLACEBO and NEUTRAL/PROCEDURAL prompts, SERIAL and PARALLEL interfaces, and direct source-provision interventions, the authors report four findings:
- An information-free instruction can improve, reduce or leave accuracy unchanged depending on model, interface and task;
- The effect is not reducible to one sentence or to parser failure (a six-pair study finds −11.5 for Qwen with all runs valid; a 30,240-run study finds ten of twelve comparisons negative, many associated with invalid tool actions);
- Successful repairs do not necessarily identify failure mechanisms. Providing correct source identifiers improves accuracy, but similar benefits also appear on clean tasks, so the improvement does not establish a disruption-specific source-selection failure.
- One targeted intervention provides stronger mechanistic evidence. In the Gemma/parallel-interface setting, directly providing the source needed after a tool failure eliminates the observed accuracy gap between the base and control conditions.
Critical strengths
- The paper draws a distinction agent-debugging work routinely ignores, and the source-hint experiment makes it concrete.
- The paper presents a reasonably systematic experimental design: The use of paired tasks, matched prompt conditions, alternative interfaces, repository-level resampling, and multiple-comparison corrections is a strength. The paper also examines action validity and tool-use trajectories rather than relying exclusively on final accuracy.
- The authors investigate whether large accuracy differences arise before the agent successfully executes a tool call. The six-pair prompt experiment is particularly useful because its reported effect persists despite valid final answers and no parser rejections. This helps avoid incorrectly interpreting every drop in end-task accuracy as a deterioration in reasoning.
Critical weaknesses
W1. The central claim of this paper is that "a successful intervention does not necessarily identify the underlying causal mechanism" is sensible, but it is also a fairly general principle of causal inference. The paper needs to establish more clearly what is specifically new about applying this principle to tool-using agents. The empirical findings primarily show that prompt effects vary across models, interfaces, and task settings. Such heterogeneity is useful to document, but heterogeneity alone is not a deep explanatory insight. For example, the paper shows that a procedural prompt can help one model and hurt another, and that changing the tool interface can substantially alter the effect. However, what generalizable insight does this provide about why these effects differ? Does the paper identify conditions under which intervention effects can be predicted or interpreted mechanistically, or does the conclusion remain that each configuration must be evaluated independently?
W2. The paper systematically varies prompt wording, model, interface, and task condition. These are appropriate experiments for estimating intervention effects, but the analysis often stops after establishing that the effects differ across conditions. A more informative study would identify and test the underlying behavioral factors that explain this variation. For example, does procedural wording alter source-selection behavior, action generation, or recovery after tool failures? Under what conditions does parallel tool access amplify or mitigate these effects? Although the paper includes some trajectory analyses, it is unclear whether these observations yield a unified, testable explanation of the results. Without stronger analysis of competing explanations, the experiments risk remaining a collection of conditional effects rather than establishing a more general account of when and why interventions work.
W3. In the 30,240-run study, the authors report 2,289 JSON-decoding failures and 124 invalid read actions. For the longest procedural prompt, Gemma fails to produce a valid first tool action in all 360 runs in each reported setting. These are important system-level failures, but they complicate the interpretation of the observed accuracy differences. In particular, an intervention that disrupts action formatting may reduce task success without providing much evidence about the underlying task-reasoning failure that the paper seeks to explain. The authors should more clearly distinguish at least three outcomes: (1) action-format validity, (2) successful tool execution and information acquisition, and (3) final-answer correctness conditional on successful execution. This decomposition would help clarify how much of the reported effect reflects changes in task-solving behavior versus failures at the interface or execution level. It would also help establish whether the proposed conclusions about mechanism identification hold when valid tool use is maintained.
W4. The evaluation focuses on relatively small open-weight models (approximately 3B–14B parameters) and controlled repository-based retrieval tasks, so it remains unclear whether the findings generalize to larger or closed models, broader software-engineering tasks, or other tool-using-agent settings.
Questions for authors
Please see weaknesses.
In particular, our findings provide three concrete observations.
First, an instruction containing no task-specific information can substantially change performance, including when every tool action is valid. The original model–interface experiments further show that the same instruction can have different effects when only the tool interface changes.
Second, a successful repair need not be specific to the failure being investigated. In the original source-identifier experiment (Section 4.2, Table 1), correct identifiers improve performance by 77.1 points for Qwen3 and 62.5 points for Ministral under disruption. However, the corresponding improvements on clean tasks are 77.1 and 64.6 points. The large improvement therefore does not establish a disruption-specific source-selection failure.
Third, the new experiments directly test how the action interface contributes to the measured intervention effect. We find that constraining the action format removes nearly all of the original prompt-related accuracy difference across seven settings and three models. Separately, the required-source replications establish a reproducible reduction in unaided recovery performance, while demonstrating that directly providing evidence can bypass the very search process whose failure is being investigated.
Together, these results go beyond observing that intervention effects vary across models or tasks. They identify particular stages at which the differences arise and test competing explanations through additional controlled experiments. They also show why improvements in final accuracy, even when substantial and reproducible, must be interpreted alongside changes in tool execution and evidence retrieval.
We also acknowledge Modrzejewski and Radzikowski (2026) as concurrent work, which appeared after our submission. Their study examines how question wording affects model comparisons under different tool conditions. Our work instead examines whether the controls and repairs used to diagnose an agent's failure support the proposed explanation within a given agent configuration.
We believe the additional evidence addresses the reviewer's two principal technical concerns. The action-format experiments establish what accounts for many of the largest accuracy differences, while the recovery replications provide stronger evidence for an effect previously supported by only 24 tasks. At the same time, we retain the limits of the constructed tasks and the single-model recovery experiment rather than drawing conclusions that the available results cannot support.