- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies when language models actually benefit from external tool calls and whether existing evaluations correctly measure those benefits. The authors introduce an eight-arm counterfactual evaluation framework to separate improvements caused by external tool information (NPIG) from changes introduced by the tool-calling protocol itself (SSE).
The paper also identifies several failure modes in tool-augmented reasoning, including premature argument generation and reduced verification after tool execution. Additionally, it investigates whether uncertainty estimates and hidden-state probes can predict when tools are needed.
Experiments across multiple models, controlled tasks, and 12 benchmarks show that requiring tool calls can hurt performance on tasks that models could otherwise solve internally. The authors also find that simple uncertainty-based routing methods struggle when tools are unreliable, while hidden-state probes fail to generalize to unseen models and benchmarks.
Critical strengths
1- The paper addresses an important question that is often overlooked in tool-augmented LLM evaluation: whether improvements actually come from external information or from changes in the interaction protocol. The proposed counterfactual framework is useful because it compares different tool responses while keeping the initial tool call fixed. This provides a more informative evaluation than simply comparing tool-augmented and closed-book accuracy.
2- The authors evaluate several models, benchmarks, and tool-calling configurations. The experiments include meaningful controls for prompt formatting, tool execution, and uncertainty-based routing. I particularly appreciate the finding that valid tool calls do not necessarily lead to better final answers, as well as the analysis showing that hidden-state probes can perform well in-distribution but fail under distribution shift. These observations provide useful insights for designing and evaluating tool-using agents.
Critical weaknesses
1- Although the counterfactual evaluation framework is useful, the underlying decomposition is largely based on differences between experimental accuracies, and the proposed entropy correction follows directly from Shannon's grouping identity. The methodological novelty therefore seems more empirical than theoretical. More importantly, the conclusions rely heavily on the selected tool-calling protocols. The main experiments use relatively restrictive action-first JSON calls and local deterministic tools, which may not represent how modern reasoning agents interact with external tools. The paper itself shows that alternative prompting and action formats can substantially change performance. Also, the proposed failure taxonomy does not fully establish the underlying mechanisms. For example, classifying an incorrect answer after a valid tool call as post-call deliberation suppression does not necessarily prove that the model stopped reasoning or verifying its answer. These concerns limit how broadly the findings can be interpreted.
2- Although the experimental evaluation is extensive, some important conclusions are based on relatively small or restricted settings. Suite II uses only 40 examples per benchmark and a generic local tool, while the visual-language findings are limited to one model family and a small number of gold-return cases. The oracle experiment also provides the correct answer directly, making its large advantage over real execution an optimistic upper bound rather than a realistic estimate of available tool improvements. Furthermore, the proposed entropy-based gate does not consistently outperform simpler uncertainty measures on actual tool benefit.
3- Almost all the main tables use extremely small fonts, making their values difficult to read without significant zooming. Many paragraphs are unnecessarily long and packed with numbers, equations, abbreviations, and statistical results. For example, the Pitfall 2 paragraph on page 8 contains numerous percentages, sample sizes, confidence intervals, and comparisons in a single dense paragraph, making the main finding unnecessarily difficult to understand. Even several table and figure captions are excessively long and contain extensive experimental details that would be better explained in the main text. The paper would benefit significantly from larger tables, shorter captions, simpler paragraphs, and a clearer separation between the main findings and supporting numerical details. The current presentation makes the paper much harder to follow than necessary.
Questions for authors
1- How well do the findings extend to more realistic tool-using agents, particularly those using native function calling, interleaved reasoning, or multi-step interactions? Since the paper shows that action formatting affects performance, additional experiments with more representative agent configurations would help establish the generality of the conclusions.
2- How do the authors distinguish actual post-call deliberation suppression from other reasons a model might produce an incorrect answer after a valid tool call? Can this mechanism be validated more directly, rather than inferring it primarily from tool-call validity and final-answer correctness?
3- How representative are the local solve-or-lookup tools of real tool-augmented applications? Also, can the authors evaluate whether the main conclusions hold with larger benchmark samples, more diverse tools, and additional VLM families? This is particularly important for the text-versus-VLM comparison in Pitfall 2.
4- Since simpler uncertainty measures perform comparably or better on actual tool-benefit prediction, what is the practical advantage of using the proposed corrected entropy? Can the authors demonstrate improvements under stronger held-out transfer settings without requiring task-specific threshold calibration?