- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies tool overuse in Tool-Integrated Reasoning: models call external tools even when their current reasoning state already suffices. The authors introduce an early-stopping estimator that, at each stopping point along a sampled trajectory, forces the model to answer without further tool calls and scores the resulting branch with the same verifier. Comparing adjacent branches gives the observable gain of each tool call and yields a four-way taxonomy — positive, negative, outcome-redundant and uncertain gain — with negative and redundant calls jointly defined as overuse. A preliminary study across six open and frontier models on four math and code benchmarks finds that overuse accounts for a large share of tool calls (82% on AMC23 for Qwen3-8B) and that far fewer calls are estimated as necessary than are actually made; the authors attribute this to an intrinsic bias toward calling tools and to poor adaptation to the utility of preceding calls.
The proposed method, GAPO, keeps DAPO's group-relative RLVR backbone and augments the advantage on the tokens of each tool-call turn with a fixed, category-specific local advantage, while the final answer turn keeps the ordinary trajectory-level advantage. The counterfactual branches are used only to build this supervision and are excluded from the actor batch, leaving the rollout distribution unchanged. Training uses Qwen3-4B/8B/32B on DAPO-Math-17K and DeepCoder. On 8B the reported average math accuracy rises from 43.71% to 57.48% while average tool calls fall from 1.91 to 0.37, with similar trends on code and at other scales. Ablations cover the local-signal weight, the redundant-call penalty, the number of branches per stopping point, and a group-relative variant of the local advantage.
Critical strengths
-
A clean, cheap operationalisation of per-call utility, evaluated at unusual breadth. The paired forced-stop branch turns "was this call worth it" into a measurable quantity across six models, four math benchmarks and a code benchmark — it converts a vague intuition into a diagnostic, and it is what makes the local advantage well-defined.
-
Simple, and evaluated against the right control. Adding a fixed per-category advantage to the tool-call token span is easy to reason about, and excluding the counterfactual branches from the actor batch correctly leaves the rollout distribution untouched. The same-reward DAPO comparison at 8B (50.22/0.83 vs. 57.48/0.37) is the strongest evidence that the turn-level signal adds something beyond plain RLVR.
Critical weaknesses
-
The baselines cannot support the central claim. GAPO is built on DAPO, so the DAPO row is an ablation of its own backbone rather than a competing method, and ReTool optimizes the opposite objective — the authors state its reward "explicitly encourages tool invocation" (§6.3) — and is reimplemented weakly enough to fall below the untrained base at 4B (34.95 vs. 44.97). The paper therefore compares against no efficiency-oriented alternative at all, and since DAPO appears only in the 8B block of Table 4 (with no explanation), "robust scalability across model capacities" rests on comparisons against untrained models.
-
The core estimator is not new. The central idea — a turn-level reward defined as the change in the likelihood of a correct answer across a tool call, combined with outcome-level supervision — has already been proposed for multi-turn agents. The author should add clear statement on core innovation points of the paper.
Questions for authors
Q1. Please add the same-reward DAPO control at 4B and 32B and explain why it was run only at 8B. Since ReTool targets accuracy rather than efficiency, please also add at least one baseline that actually optimizes tool-call efficiency. As it stands there is a single controlled comparison, at a single scale. It's also advisable to add other tool-efficiency works as baselines.
Q2. Table 3's estimated minimum is 0.00 for Qwen3-4B on both AMC23 and AIME24, which I cannot reconcile with Table 4. The earliest correct stopping point is zero exactly when the branch taken before any tool call already answers correctly, so an average of 0.00 implies a near-100% single-sample no-tool success rate for a 4B model on these benchmarks — against the 85.50% and 60.48% reported for the same model and settings. Please clarify: (a) does a correctness label mean "the model answered correctly once" or "the model is reliably capable"? (b) How are trajectories with no correct stopping point handled — excluded, assigned the full trajectory length, or assigned zero? If dropped or mapped to zero, the Minimum column is biased downward, and since the Actual column reproduces Table 4 exactly, the two columns would be averaged over different trajectory sets.
Q3. IGPO (arXiv:2510.14967) and TIPS (arXiv:2603.22293) already define turn-level rewards as the change in the likelihood of a correct answer across a tool call and combine them with outcome-level supervision; neither is cited. What is new here beyond estimating that same quantity from a single sampled verifier label and discretizing it into four categories?
4 comments
Author Response for Q2 & Q3
Response for Q2
Thank you for raising this important concern. We clarify the two aspects as follows.
(a) A correctness label simply indicates whether the model answered correctly once in a sampled early-stopping completion.
(b) All trajectories are retained in the calculation. If a correct early-stopping branch is found, we use its earliest stopping position; otherwise, we assign the trajectory its original number of tool calls (N_i).
This explains the 0.00 values for Qwen3-4B on AMC23 and AIME24. Qwen3-4B has a relatively low tendency to invoke tools, resulting in many original trajectories without any tool calls. For these trajectories, the estimated minimum is zero regardless of whether their final answers are correct or incorrect, simply because no tool calls were made. Thus, the reported 0.00 does not imply a near-100% zero-tool accuracy. It primarily reflects the model's low tool-calling frequency and the definition of our estimator.
Response for Q3
Thank you for pointing out these relevant works. We agree that IGPO and TIPS have explored similar turn-level supervision combined with outcome-level rewards, and we will add both citations and discuss their relationship to our work in the revised manuscript.
Our work differs primarily in its perspective and objective. Rather than focusing on answer-likelihood improvement, we investigate tool overuse, i.e., whether a tool call is actually necessary for solving a task. Specifically, even if a tool call increases the likelihood of a correct answer, if the model can already produce a correct answer both before and after the call , GAPO identifies it as an outcome-redundant call and assigns it a negative local advantage. This allows us to explicitly discourage unnecessary tool calls while reinforcing genuinely beneficial ones.
Therefore, our main contribution lies in systematically characterizing tool overuse and developing a targeted optimization method to reduce redundant and harmful tool interactions. Across multiple mathematical reasoning 、code-generation and Agentic search benchmarks, GAPO achieves consistently strong performance, outperforming several competitive tool-use baselines while substantially reducing unnecessary tool calls.
Thank you for your thoughtful response. I believe the additional experiments you have provided sufficiently address the weaknesses in the paper’s experiments. The paper now appears solid and complete. However, I still find its level of innovation to be limited, and it fails to offer any particularly valuable insights.
Overall, although the quality of the paper has clearly improved following the inclusion of these additional experiments, the concerns regarding its innovation lead me to categorize the paper as "borderline." Given the 4-point scoring scale, I regret to say that I will maintain my original score.
Response to Reviewer
We thank the reviewer for their thoughtful comments regarding the novelty of our work. We understand the concern that GAPO may represent only an incremental improvement over existing tool-use optimization methods. However, we believe that GAPO differs substantially from prior work in both its methodological design and optimization mechanism. Although several recent studies have investigated when and whether large language models (LLMs) should invoke external tools, the supervision signals and optimization mechanisms adopted by these methods differ significantly from ours.
The key innovation of GAPO lies in explicitly estimating the observable marginal contribution of each tool call through paired early-stopping branches and translating this information into fine-grained local advantages for policy optimization. Specifically, IGPO primarily measures the utility of tool calls based on changes in the policy's probability of generating a correct answer, whereas TIPS relies on a teacher model's likelihood estimates of candidate answers. In contrast, GAPO directly evaluates the actual effect of tool calls based on answer correctness as assessed by a verifier. This design enables the optimization process to more directly identify tool calls that genuinely improve answer correctness and distinguish beneficial tool interactions from unnecessary or redundant ones, without relying on likelihood estimates from an additional teacher model. Therefore, GAPO addresses tool overuse from a more direct, outcome-grounded perspective.
Our experimental results further demonstrate the effectiveness of this design. Beyond mathematical reasoning, we also evaluate GAPO on code generation and agentic search tasks, where it consistently achieves performance improvements. Specifically, on Qwen2.5-3B and Qwen2.5-7B, GAPO improves average Pass@1 over EAPO by 2.43 and 5.27 percentage points, respectively, while reducing the average number of tool calls by 19.1% and 20.6%, respectively. On agentic search benchmarks, GAPO achieves the highest average F1 and EM scores among all compared methods at both Qwen2.5 model scales, while making fewer tool calls than EAPO and AutoSearch. These results demonstrate that the methodological differences of GAPO are not merely conceptual, but also translate into tangible performance and efficiency gains across different task types and model scales.
Furthermore, following the reviewer's suggestion, we have expanded the Related Work section and incorporated the aforementioned experimental comparisons into the revised manuscript. These additions aim to more clearly articulate the methodological distinctions between GAPO and existing approaches, as well as demonstrate its effectiveness across different task settings.
Author Response for Q1
Thank you for this valuable suggestion. We agree that the original baseline comparisons were insufficient. We have conducted additional experiments to address the concerns regarding model scales, efficiency-oriented baselines, and task diversity.
1. Extending the same-reward DAPO control. We initially reported DAPO only on Qwen3-8B because our goal was to isolate the effect of GAPO's turn-level supervision from the influence of ReTool's explicit tool-call reward. We selected Qwen3-8B as a representative model for this controlled comparison. Following the reviewer's suggestion, we have extended the same-reward DAPO experiments to Qwen3-4B and Qwen3-32B. The results are reported below (Accuracy / Tool Calls).
2. Adding tool-efficiency-oriented baselines.
To address the lack of efficiency-oriented comparisons, we introduce three additional mathematical reasoning baselines: (i) DAPO (No-Tool), which removes tool access; (ii) DAPO (Reward Shaping), which applies explicit tool-use penalties; and (iii) EAPO, which specifically targets selective tool use. We also introduce AutoSearch for agentic search comparisons. To enable more controlled comparisons with EAPO and AutoSearch, we additionally reproduce GAPO on Qwen2.5-3B/7B-Instruct and report Pass@1 and tool-call numbers on mathematical reasoning benchmarks. The results are summarized below.
Mathematical Reasoning (Pass@1 / Tool Calls)
GAPO achieves the best average Pass@1 at both model scales, outperforming EAPO by 2.43 and 5.27 percentage points, respectively, while reducing tool calls by 19.1% and 20.6%. It also outperforms the no-tool and reward-shaping baselines, demonstrating the benefit of fine-grained tool-call supervision.
Agentic Search (F1 / EM / Tool Calls)
On agentic search, GAPO achieves the best average F1 and EM among the compared methods on both Qwen2.5 scales, while using fewer tool calls than EAPO and AutoSearch. It also consistently outperforms DAPO on Qwen3-4B/8B in both answer quality and tool-use efficiency.
Overall, these additional experiments provide broader comparisons across model scales, baseline objectives, and task domains. They demonstrate that GAPO improves task performance while reducing unnecessary tool interactions, outperforming both general RL baselines and methods specifically designed for selective tool use. We will incorporate these results and the corresponding discussions into the revised manuscript.