acceptodds
Under review as a conference paper at ICLR 2027

Not Every Tool Call Helps: Fine-Grained Credit Assignment for Efficient Tool-Integrated Reasoning

Abstract

Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to solve complex reasoning problems with external tools, yet existing reinforcement learning methods largely rely on coarse-grained trajectory-level rewards and overlook the utility of individual tool calls. We systematically study tool-use behavior and identify pervasive tool overuse: models often invoke tools even when their current reasoning states are already sufficient for correct solutions. To quantify this behavior, we introduce an early-stopping estimator that measures the observable correctness gain of each tool call by comparing outcomes immediately before and after invocation. Our analysis reveals that many tool calls are redundant or harmful, driven by persistent tool-calling bias and limited adaptation to prior tool utility. Based on these findings, we propose GAPO (Gain-Aware Policy Optimization), which converts tool-call gains into fine-grained local advantages and combines them with trajectory-level supervision to optimize tool-calling behavior. Across mathematical reasoning and code-generation benchmarks with Qwen3 models from 4B to 32B, GAPO consistently improves reasoning performance and tool-use efficiency. On Qwen3-8B, it improves average mathematical reasoning accuracy by 13.77 percentage points while reducing tool calls by 80.63%.

Primary area:
reinforcement learning
Keywords:
Tool-Integrated Reasoning, Tool calls

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UWJvOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies tool overuse in Tool-Integrated Reasoning: models call external tools even when their current reasoning state already suffices. The authors introduce an early-stopping estimator that, at each stopping point along a sampled trajectory, forces the model to answer without further tool calls and scores the resulting branch with the same verifier. Comparing adjacent branches gives the observable gain of each tool call and yields a four-way taxonomy — positive, negative, outcome-redundant and uncertain gain — with negative and redundant calls jointly defined as overuse. A preliminary study across six open and frontier models on four math and code benchmarks finds that overuse accounts for a large share of tool calls (82% on AMC23 for Qwen3-8B) and that far fewer calls are estimated as necessary than are actually made; the authors attribute this to an intrinsic bias toward calling tools and to poor adaptation to the utility of preceding calls.

The proposed method, GAPO, keeps DAPO's group-relative RLVR backbone and augments the advantage on the tokens of each tool-call turn with a fixed, category-specific local advantage, while the final answer turn keeps the ordinary trajectory-level advantage. The counterfactual branches are used only to build this supervision and are excluded from the actor batch, leaving the rollout distribution unchanged. Training uses Qwen3-4B/8B/32B on DAPO-Math-17K and DeepCoder. On 8B the reported average math accuracy rises from 43.71% to 57.48% while average tool calls fall from 1.91 to 0.37, with similar trends on code and at other scales. Ablations cover the local-signal weight, the redundant-call penalty, the number of branches per stopping point, and a group-relative variant of the local advantage.

Critical strengths
  1. A clean, cheap operationalisation of per-call utility, evaluated at unusual breadth. The paired forced-stop branch turns "was this call worth it" into a measurable quantity across six models, four math benchmarks and a code benchmark — it converts a vague intuition into a diagnostic, and it is what makes the local advantage well-defined.

  2. Simple, and evaluated against the right control. Adding a fixed per-category advantage to the tool-call token span is easy to reason about, and excluding the counterfactual branches from the actor batch correctly leaves the rollout distribution untouched. The same-reward DAPO comparison at 8B (50.22/0.83 vs. 57.48/0.37) is the strongest evidence that the turn-level signal adds something beyond plain RLVR.

Critical weaknesses
  1. The baselines cannot support the central claim. GAPO is built on DAPO, so the DAPO row is an ablation of its own backbone rather than a competing method, and ReTool optimizes the opposite objective — the authors state its reward "explicitly encourages tool invocation" (§6.3) — and is reimplemented weakly enough to fall below the untrained base at 4B (34.95 vs. 44.97). The paper therefore compares against no efficiency-oriented alternative at all, and since DAPO appears only in the 8B block of Table 4 (with no explanation), "robust scalability across model capacities" rests on comparisons against untrained models.

  2. The core estimator is not new. The central idea — a turn-level reward defined as the change in the likelihood of a correct answer across a tool call, combined with outcome-level supervision — has already been proposed for multi-turn agents. The author should add clear statement on core innovation points of the paper.

Questions for authors

Q1. Please add the same-reward DAPO control at 4B and 32B and explain why it was run only at 8B. Since ReTool targets accuracy rather than efficiency, please also add at least one baseline that actually optimizes tool-call efficiency. As it stands there is a single controlled comparison, at a single scale. It's also advisable to add other tool-efficiency works as baselines.

Q2. Table 3's estimated minimum is 0.00 for Qwen3-4B on both AMC23 and AIME24, which I cannot reconcile with Table 4. The earliest correct stopping point is zero exactly when the branch taken before any tool call already answers correctly, so an average of 0.00 implies a near-100% single-sample no-tool success rate for a 4B model on these benchmarks — against the 85.50% and 60.48% reported for the same model and settings. Please clarify: (a) does a correctness label mean "the model answered correctly once" or "the model is reliably capable"? (b) How are trajectories with no correct stopping point handled — excluded, assigned the full trajectory length, or assigned zero? If dropped or mapped to zero, the Minimum column is biased downward, and since the Actual column reproduces Table 4 exactly, the two columns would be averaged over different trajectory sets.

Q3. IGPO (arXiv:2510.14967) and TIPS (arXiv:2603.22293) already define turn-level rewards as the change in the likelihood of a correct answer across a tool call and combine them with outcome-level supervision; neither is cited. What is new here beyond estimating that same quantity from a single sampled verifier label and discretizing it into four categories?

4 comments
AuthorsOpenReview ↗

Author Response for Q1

Thank you for this valuable suggestion. We agree that the original baseline comparisons were insufficient. We have conducted additional experiments to address the concerns regarding model scales, efficiency-oriented baselines, and task diversity.

1. Extending the same-reward DAPO control. We initially reported DAPO only on Qwen3-8B because our goal was to isolate the effect of GAPO's turn-level supervision from the influence of ReTool's explicit tool-call reward. We selected Qwen3-8B as a representative model for this controlled comparison. Following the reviewer's suggestion, we have extended the same-reward DAPO experiments to Qwen3-4B and Qwen3-32B. The results are reported below (Accuracy / Tool Calls).

ModelAMC23AIME24AIME25HMMT25AMOAVG.
Qwen3-4B + DAPO88.75/0.1763.33/0.3455.83/0.3527.08/0.137.25/0.2348.45/0.24
Qwen3-32B + DAPO85.63/1.8273.75/1.6658.75/1.9029.58/1.5012.75/2.5752.09/1.89

2. Adding tool-efficiency-oriented baselines.

To address the lack of efficiency-oriented comparisons, we introduce three additional mathematical reasoning baselines: (i) DAPO (No-Tool), which removes tool access; (ii) DAPO (Reward Shaping), which applies explicit tool-use penalties; and (iii) EAPO, which specifically targets selective tool use. We also introduce AutoSearch for agentic search comparisons. To enable more controlled comparisons with EAPO and AutoSearch, we additionally reproduce GAPO on Qwen2.5-3B/7B-Instruct and report Pass@1 and tool-call numbers on mathematical reasoning benchmarks. The results are summarized below.

Mathematical Reasoning (Pass@1 / Tool Calls)

ModelMethodAIME24AIME25Math500Avg.
Qwen2.5-3BTIR10.0/0.236.70/0.1053.2/0.8523.3/0.39
DAPO (No-Tool)16.7/-10.0/-72.0/-32.9/-
DAPO (Reward Shaping)20.0/0.7210.0/0.7770.8/0.9533.6/0.81
DAPO (Vanilla)16.7/1.2113.3/1.3572.2/1.3034.07/1.29
EAPO23.3/0.9013.3/0.8970.1/1.0335.57/0.94
GAPO26.7/0.7413.3/0.7274.0/0.8338.00/0.76
Qwen2.5-7BTIR10.0/0.2010.0/0.1969.0/0.4729.7/0.29
DAPO (No-Tool)23.3/-23.3/-76.8/-41.1/-
DAPO (Reward Shaping)26.7/0.7920.0/0.8277.0/0.8841.2/0.83
DAPO (Vanilla)26.7/1.0223.3/1.1178.4/1.0942.80/1.07
EAPO30.0/1.0026.6/0.9378.8/0.9845.13/0.97
GAPO40.0/0.8130.0/0.7281.2/0.7750.40/0.77

GAPO achieves the best average Pass@1 at both model scales, outperforming EAPO by 2.43 and 5.27 percentage points, respectively, while reducing tool calls by 19.1% and 20.6%. It also outperforms the no-tool and reward-shaping baselines, demonstrating the benefit of fine-grained tool-call supervision.

Agentic Search (F1 / EM / Tool Calls)

ModelMethodHotpotQA2WikiBamboogleAvg.
Qwen2.5-3BTIR16.1/10.7/1.4314.8/9.5/1.7216.3/11.2/1.3715.7/10.47/1.51
EAPO46.5/38.5/1.8452.4/45.5/2.0448.0/40.0/1.9048.97/41.33/1.93
AutoSearch50.5/39.4/1.4947.7/41.6/1.6944.7/33.1/1.8347.63/38.0/1.67
GAPO51.2/39.9/1.3251.9/45.4/1.6948.7/41.6/1.7750.6/42.3/1.59
Qwen2.5-7BTIR14.1/6.8/2.1113.3/5.3/2.4511.2/7.3/2.3512.87/6.5/2.30
EAPO54.1/42.5/1.9958.6/51.5/1.7160.4/48.8/2.1757.7/47.6/1.96
AutoSearch54.2/42.7/1.3450.3/43.6/1.5854.9/41.9/1.5353.1/42.7/1.48
GAPO55.3/43.1/1.2057.2/53.0/1.4461.2/49.6/1.4957.9/48.6/1.38
Qwen3-4BTIR54.5/31.1/1.6955.6/33.7/1.8569.8/47.2/2.0160.0/37.3/1.85
DAPO67.3/45.2/1.8359.8/52.6/1.7772.4/55.2/2.1366.5/51.0/1.91
GAPO67.9/46.5/1.4462.3/53.4/1.5273.6/56.8/1.8767.9/52.2/1.61
Qwen3-8BTIR59.5/31.0/2.8158.1/35.9/2.3372.6/48.0/3.0163.4/38.3/2.72
DAPO68.6/50.7/2.9260.1/54.8/2.5378.1/56.0/3.5168.9/53.8/2.99
GAPO70.1/52.1/2.1562.0/55.7/2.0379.4/59.2/2.3370.5/55.7/2.17

On agentic search, GAPO achieves the best average F1 and EM among the compared methods on both Qwen2.5 scales, while using fewer tool calls than EAPO and AutoSearch. It also consistently outperforms DAPO on Qwen3-4B/8B in both answer quality and tool-use efficiency.

Overall, these additional experiments provide broader comparisons across model scales, baseline objectives, and task domains. They demonstrate that GAPO improves task performance while reducing unnecessary tool interactions, outperforming both general RL baselines and methods specifically designed for selective tool use. We will incorporate these results and the corresponding discussions into the revised manuscript.

AuthorsOpenReview ↗

Author Response for Q2 & Q3

Response for Q2

Thank you for raising this important concern. We clarify the two aspects as follows.

(a) A correctness label simply indicates whether the model answered correctly once in a sampled early-stopping completion.

(b) All trajectories are retained in the calculation. If a correct early-stopping branch is found, we use its earliest stopping position; otherwise, we assign the trajectory its original number of tool calls (N_i).

This explains the 0.00 values for Qwen3-4B on AMC23 and AIME24. Qwen3-4B has a relatively low tendency to invoke tools, resulting in many original trajectories without any tool calls. For these trajectories, the estimated minimum is zero regardless of whether their final answers are correct or incorrect, simply because no tool calls were made. Thus, the reported 0.00 does not imply a near-100% zero-tool accuracy. It primarily reflects the model's low tool-calling frequency and the definition of our estimator.

Response for Q3

Thank you for pointing out these relevant works. We agree that IGPO and TIPS have explored similar turn-level supervision combined with outcome-level rewards, and we will add both citations and discuss their relationship to our work in the revised manuscript.

Our work differs primarily in its perspective and objective. Rather than focusing on answer-likelihood improvement, we investigate tool overuse, i.e., whether a tool call is actually necessary for solving a task. Specifically, even if a tool call increases the likelihood of a correct answer, if the model can already produce a correct answer both before and after the call , GAPO identifies it as an outcome-redundant call and assigns it a negative local advantage. This allows us to explicitly discourage unnecessary tool calls while reinforcing genuinely beneficial ones.

Therefore, our main contribution lies in systematically characterizing tool overuse and developing a targeted optimization method to reduce redundant and harmful tool interactions. Across multiple mathematical reasoning 、code-generation and Agentic search benchmarks, GAPO achieves consistently strong performance, outperforming several competitive tool-use baselines while substantially reducing unnecessary tool calls.

Thank you for your thoughtful response. I believe the additional experiments you have provided sufficiently address the weaknesses in the paper’s experiments. The paper now appears solid and complete. However, I still find its level of innovation to be limited, and it fails to offer any particularly valuable insights.

Overall, although the quality of the paper has clearly improved following the inclusion of these additional experiments, the concerns regarding its innovation lead me to categorize the paper as "borderline." Given the 4-point scoring scale, I regret to say that I will maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for their thoughtful comments regarding the novelty of our work. We understand the concern that GAPO may represent only an incremental improvement over existing tool-use optimization methods. However, we believe that GAPO differs substantially from prior work in both its methodological design and optimization mechanism. Although several recent studies have investigated when and whether large language models (LLMs) should invoke external tools, the supervision signals and optimization mechanisms adopted by these methods differ significantly from ours.

The key innovation of GAPO lies in explicitly estimating the observable marginal contribution of each tool call through paired early-stopping branches and translating this information into fine-grained local advantages for policy optimization. Specifically, IGPO primarily measures the utility of tool calls based on changes in the policy's probability of generating a correct answer, whereas TIPS relies on a teacher model's likelihood estimates of candidate answers. In contrast, GAPO directly evaluates the actual effect of tool calls based on answer correctness as assessed by a verifier. This design enables the optimization process to more directly identify tool calls that genuinely improve answer correctness and distinguish beneficial tool interactions from unnecessary or redundant ones, without relying on likelihood estimates from an additional teacher model. Therefore, GAPO addresses tool overuse from a more direct, outcome-grounded perspective.

Our experimental results further demonstrate the effectiveness of this design. Beyond mathematical reasoning, we also evaluate GAPO on code generation and agentic search tasks, where it consistently achieves performance improvements. Specifically, on Qwen2.5-3B and Qwen2.5-7B, GAPO improves average Pass@1 over EAPO by 2.43 and 5.27 percentage points, respectively, while reducing the average number of tool calls by 19.1% and 20.6%, respectively. On agentic search benchmarks, GAPO achieves the highest average F1 and EM scores among all compared methods at both Qwen2.5 model scales, while making fewer tool calls than EAPO and AutoSearch. These results demonstrate that the methodological differences of GAPO are not merely conceptual, but also translate into tangible performance and efficiency gains across different task types and model scales.

Furthermore, following the reviewer's suggestion, we have expanded the Related Work section and incorporated the aforementioned experimental comparisons into the revised manuscript. These additions aim to more clearly articulate the methodological distinctions between GAPO and existing approaches, as well as demonstrate its effectiveness across different task settings.

Reviewer_UqgeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies tool overuse in LLMs: models often call external tools even when their current information is enough to solve the problem. The authors estimate the benefit of each tool call by asking the model to answer from the states before and after the call, without using any additional tools. Based on whether these answers are correct, they classify calls as beneficial, harmful, redundant, or uncertain. They then propose Gain-Aware Policy Optimization (GAPO), which combines feedback on individual tool calls with rewards for the final answer. Experiments on mathematical reasoning and code generation tasks using Qwen3 models show that GAPO improves accuracy while reducing tool use.

Critical strengths
  1. GAPO offers a practical solution to tool-level credit assignment. It combines feedback on individual tool calls with rewards for the final answer, helping the model learn which calls are useful. In the 8B math experiments, it achieves higher accuracy with fewer tool calls than standard DAPO.
  2. The experimental analysis provides useful insights into tool overuse. The paper shows that models often continue using tools even when they can already produce a correct answer, and that tool use does not consistently match problem difficulty. These findings help explain why final-answer rewards alone may provide insufficient guidance for tool use. The analysis is valuable beyond GAPO because it can inform the design and evaluation of other models that use tools.
Critical weaknesses

The experiments do not clearly establish the added value of judging whether each tool call helps. GAPO rewards useful calls and penalizes redundant or harmful ones. However, a simpler method could also reduce unnecessary tool use: give a small penalty for every tool call while still rewarding correct final answers. The paper lacks a direct comparison with this simpler method under the same training conditions, with the penalty strength properly tuned.

Estimating the value of individual calls is a central contribution of the paper, and it requires extra sampled answers during training. Comparing accuracy and tool-call counts against a matched, uniform-penalty baseline would help establish whether this additional step provides a meaningful benefit. Without that comparison, the evidence supporting this central design choice remains incomplete.

Questions for authors

How much does GAPO gain from distinguishing useful and unnecessary calls, beyond the benefit of generally discouraging tool use?

2 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for this insightful comment. We agree that distinguishing the benefits of fine-grained tool-call credit assignment from simply discouraging tool use is important for validating GAPO.

To address this concern, we conduct additional experiments on Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct, introducing three mathematical reasoning baselines: (1) DAPO (No-Tool), which removes tool access; (2) DAPO (Reward Shaping), which applies a uniform penalty to tool calls; and (3) EAPO, which explicitly targets selective tool use. We also include vanilla DAPO for comparison. The results are reported in terms of average Pass@1 / tool calls across AIME24, AIME25, and Math500.

ModelMethodPass@1 (%)Tool Calls
Qwen2.5-3BDAPO (Vanilla)34.071.29
DAPO (Reward Shaping)33.600.81
EAPO35.570.94
GAPO38.000.76
Qwen2.5-7BDAPO (Vanilla)42.801.07
DAPO (Reward Shaping)41.200.83
EAPO45.130.97
GAPO50.400.77

The results highlight two important observations.

First, compared with the uniform-penalty baseline, GAPO improves average Pass@1 by 4.40 and 9.20 percentage points on Qwen2.5-3B and Qwen2.5-7B, respectively, while using slightly fewer tool calls (0.76 vs. 0.81 and 0.77 vs. 0.83). This indicates that GAPO's gains cannot be explained merely by discouraging tool use.

Second, GAPO also outperforms EAPO, achieving higher accuracy with fewer tool calls at both model scales. These results support the importance of explicitly estimating individual tool-call utility rather than applying a uniform penalty to all tool interactions.

Regarding your question about the additional benefits of distinguishing useful from unnecessary tool calls beyond generally discouraging tool use, GAPO achieves Pass@1 improvements of 4.40 and 9.20 percentage points over the uniform-penalty baseline on Qwen2.5-3B and Qwen2.5-7B, respectively, while making slightly fewer tool calls. These results suggest that GAPO's advantage stems not merely from reducing tool usage, but from learning to distinguish beneficial tool calls from redundant or harmful ones, thereby achieving a better balance between reasoning accuracy and tool-use efficiency. We will incorporate these results and corresponding discussions into the revised manuscript.

Thank you for the response. I will keep my positive score.

Reviewer_K5upOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies tool overuse problem in tool-integrated reasoning. The overuse is defined as: models invoke external tools even when their current reasoning state is already enough to produce a correct answer. To measure the usefulness of the tool calls, the authors propose an early-stopping estimator. At the states immediately before and after each tool call, a forced-stop instruction is appended, and the model must answer without further tool use. Building on this, the paper proposes GAPO, a DAPO-based method. GAPO is evaluated on several math and code benchmarks. The results show that GAPO improves accuracy while substantially reducing tool calls.

Critical strengths
  1. The paper is clear and well-written. The problem formulation, the early-stopping estimator, the gain taxonomy and the advantage assignment are all defined explicitly, making the method easy to follow and understand.
  2. The early-stopping estimator is simple, intuitive, and very effective.
  3. The experiments are comprehensive and several designs are supported by ablation studies.
Critical weaknesses
  1. My main concern is the novelty and insufficient comparison with previous work on tool overuse. The paper presents tool overuse as a phenomenon it identifies and characterizes, but this problem has already been studied. For example, SMART (Qian et al., Findings of ACL 2025) explicitly frames and mitigates tool overuse; Tool-Light (Chen et al., ICLR 2026) targets both tool overuse and underuse in TIR. In agentic search, over-search is also an active research topic, e.g., AutoSearch (Sun et al., Findings of ACL 2026) and the very recent SAAS (Tang et al., 2026). The core technical idea is also similar to some previous studies. The idea is probing whether the current state already suffices to answer and to derive turn-level credit from the change. For example, AutoSearch has the agent produce an intermediate answer at each step. It takes the earliest correct step as the minimal sufficient depth and penalizes all subsequent steps.
  2. The baselines are also insufficient. For math, the only TIR baseline is ReTool. Several important comparisons are missing, including methods designed to reduce tool overuse, simple reward-shaping baselines, and no-tool RL baseline.
  3. Both the task type and tool scope are very limited. All experiments use a single tool (a Python interpreter) on math and code tasks. I would suggest evaluation on some of the following tasks: (1) Knowledge-intensive search tasks (NQ, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle); (2) Multi-tool and function-calling settings (BFCL, τ2\tau^2-bench, When2Call); (3) Interactive agentic environments such as AppWorld and Terminal-Bench.
  4. The training overhead is not well-discussed. Each trajectory requires additional forced-stop branches. Since efficiency is the main selling point of the paper, the authors should also report the additional rollout tokens and wall-clock time relative to DAPO and ReTool. They should discuss how this overhead scales to agentic tasks with tens or hundreds of tool calls.

Minor problems:

  1. No standard deviations or significance tests are reported across the 8 runs.
  2. Section 3.2 attributes GRPO to DeepSeek-R1 (2025), but it was introduced in DeepSeekMath (2024).
Questions for authors

Please check my concerns in weaknesses.

4 comments
AuthorsOpenReview ↗

Author Response for Weakness 1

We thank the reviewer for highlighting prior work on tool overuse. We agree that tool overuse and over-search have been studied, and will revise the manuscript to acknowledge these contributions and clarify our novelty.

Our novelty

Our contribution is not the first identification of tool overuse, but an analysis of each tool call’s marginal correctness gain and its causes from a new perspective, including persistent calling bias and poor adaptation to the utility of previous calls. Building on these insights, we propose GAPO, which derives fine-grained tool-call advantages through paired early-stopping branches to improve credit assignment and selective tool use.

Prior work addresses tool overuse through rationale-based SFT (SMART; Qian et al., 2025), entropy-guided preference learning (Tool-Light; Chen et al., 2026), tool-free rollouts (EAPO; Chen et al., 2026), and search-boundary estimation (SAAS; Tang et al., 2026). Most related is AutoSearch (Sun et al., 2026), which identifies sufficient search depth using intermediate answers. In contrast, GAPO evaluates each tool call’s marginal correctness gain and targets general tool-integrated reasoning.

Adding more baselines.

To address the lack of comparisons, we introduce three additional mathematical reasoning baselines: (i) DAPO (No-Tool), which removes tool access; (ii) DAPO (Reward Shaping), which applies explicit tool-use penalties; and (iii) EAPO, which specifically targets selective tool use. We also introduce AutoSearch for agentic search comparisons. To enable more controlled comparisons with EAPO and AutoSearch, we additionally reproduce GAPO on Qwen2.5-3B/7B-Instruct and report Pass@1 and tool-call numbers on mathematical reasoning benchmarks. The results are summarized below.

Mathematical Reasoning (Pass@1 / Tool Calls)

ModelMethodAIME24AIME25Math500Avg.
Qwen2.5-3BTIR10.0/0.236.70/0.1053.2/0.8523.3/0.39
DAPO (No-Tool)16.7/-10.0/-72.0/-32.9/-
DAPO (Reward Shaping)20.0/0.7210.0/0.7770.8/0.9533.6/0.81
DAPO (Vanilla)16.7/1.2113.3/1.3572.2/1.3034.07/1.29
EAPO23.3/0.9013.3/0.8970.1/1.0335.57/0.94
GAPO26.7/0.7413.3/0.7274.0/0.8338.00/0.76
Qwen2.5-7BTIR10.0/0.2010.0/0.1969.0/0.4729.7/0.29
DAPO (No-Tool)23.3/-23.3/-76.8/-41.1/-
DAPO (Reward Shaping)26.7/0.7920.0/0.8277.0/0.8841.2/0.83
DAPO (Vanilla)26.7/1.0223.3/1.1178.4/1.0942.80/1.07
EAPO30.0/1.0026.6/0.9378.8/0.9845.13/0.97
GAPO40.0/0.8130.0/0.7281.2/0.7750.40/0.77

GAPO achieves the best average Pass@1 at both model scales, outperforming EAPO by 2.43 and 5.27 percentage points, respectively, while reducing tool calls by 19.1% and 20.6%. It also outperforms the no-tool and reward-shaping baselines, demonstrating the benefit of fine-grained tool-call supervision.

Agentic Search (F1 / EM / Tool Calls)

ModelMethodHotpotQA2WikiBamboogleAvg.
Qwen2.5-3BTIR16.1/10.7/1.4314.8/9.5/1.7216.3/11.2/1.3715.7/10.47/1.51
EAPO46.5/38.5/1.8452.4/45.5/2.0448.0/40.0/1.9048.97/41.33/1.93
AutoSearch50.5/39.4/1.4947.7/41.6/1.6944.7/33.1/1.8347.63/38.0/1.67
GAPO51.2/39.9/1.3251.9/45.4/1.6948.7/41.6/1.7750.6/42.3/1.59
Qwen2.5-7BTIR14.1/6.8/2.1113.3/5.3/2.4511.2/7.3/2.3512.87/6.5/2.30
EAPO54.1/42.5/1.9958.6/51.5/1.7160.4/48.8/2.1757.7/47.6/1.96
AutoSearch54.2/42.7/1.3450.3/43.6/1.5854.9/41.9/1.5353.1/42.7/1.48
GAPO55.3/43.1/1.2057.2/53.0/1.4461.2/49.6/1.4957.9/48.6/1.38
Qwen3-4BTIR54.5/31.1/1.6955.6/33.7/1.8569.8/47.2/2.0160.0/37.3/1.85
DAPO67.3/45.2/1.8359.8/52.6/1.7772.4/55.2/2.1366.5/51.0/1.91
GAPO67.9/46.5/1.4462.3/53.4/1.5273.6/56.8/1.8767.9/52.2/1.61
Qwen3-8BTIR59.5/31.0/2.8158.1/35.9/2.3372.6/48.0/3.0163.4/38.3/2.72
DAPO68.6/50.7/2.9260.1/54.8/2.5378.1/56.0/3.5168.9/53.8/2.99
GAPO70.1/52.1/2.1562.0/55.7/2.0379.4/59.2/2.3370.5/55.7/2.17

On agentic search, GAPO achieves the best average F1 and EM among the compared methods on both Qwen2.5 scales, while using fewer tool calls than EAPO and AutoSearch. It also consistently outperforms DAPO on Qwen3-4B/8B in both answer quality and tool-use efficiency.

Overall, these additional experiments provide broader comparisons across model scales, baseline objectives, and task domains. They demonstrate that GAPO improves task performance while reducing tool interactions, outperforming both general RL baselines and methods specifically designed for selective tool use. We will incorporate these results and the corresponding discussions into the revised manuscript.

AuthorsOpenReview ↗

Author Response for Weakness 2-4

Response for Weakness 2

We additionally supplement three baselines: (i) DAPO (No-Tool), which removes tool access; (ii) DAPO (Reward Shaping), which applies explicit tool-use penalties; and (iii) EAPO, which specifically targets selective tool use. These additions are reported in the "Author Response for Weakness 1".

Response for Weakness 3

We additionally include evaluations on HotpotQA, 2WikiMultiHopQA, and Bamboogle, as reported in the "Author Response for Weakness 1".

Response for Weakness 4

We thank the reviewer for highlighting the training overhead of GAPO.

Theoretical Overhead For a trajectory containing NN tool calls, the original rollout consists of N+1N+1 model responses (including tool-call turns and the final answer). GAPO additionally generates N+1N+1 unique early-stopping branches, with adjacent tool-call comparisons sharing intermediate branches. Therefore, with M=1M=1, the total number of response-generation rounds is approximately 2× that of DAPO, assuming the same number of tool calls.

However, this is a response-count-based estimate rather than a direct measure of wall-clock time. In practice, GAPO progressively reduces tool calls during training, thereby reducing the number of required branches. Moreover, early-stopping branches are excluded from the actor-training batch and incur no additional actor backpropagation.

Empirical Overhead We measure the per-step training time (seconds/step) and average peak accelerator memory utilization under identical configurations. All experiments use a global batch size of 64, a mini-batch size of 8, and 8 rollouts per prompt, running on 64 Ascend 910B NPUs across 8 compute nodes (sequence parallel size 8).

ModelMethodTime (s/step)Peak Memory (%)Time Ratio
Qwen3-4BDAPO887.8277.741.00×
GAPO1142.3683.861.29×
Qwen3-8BDAPO661.2578.991.00×
GAPO948.7379.051.43×
Qwen3-32BDAPO1110.789.491.00×
GAPO1715.0988.471.54×

GAPO incurs a 28.7%–54.4% increase in per-step training time, remaining below 1.55× DAPO across all model scales. Importantly, its peak memory utilization remains largely comparable to DAPO.

Scalability For long-horizon agentic tasks involving tens or hundreds of tool calls, the number of early-stopping branches grows linearly as O(N)O(N), rather than quadratically or exponentially. Although this introduces additional rollout costs for long trajectories, the sampling size (MM) provides a direct control over this overhead.

We will clarify this computational trade-off and discuss its implications for long-horizon agentic tasks in the revised manuscript.

AuthorsOpenReview ↗

Author Response for Minor problems

1. No standard deviations or significance tests are reported across the 8 runs.

Thank you for this helpful suggestion. We agree that reporting only mean values does not sufficiently reflect experimental variability. In the revised manuscript, we will report the mean and standard deviation (Mean ± Std) across eight independent evaluation runs. The standard deviations for the main experimental results of Qwen3-8B are provided below. We will also include the corresponding standard deviations for the remaining experimental results in the revised manuscript.

MethodMath Avg. Acc. (%)Math Avg. Tool Calls
Qwen3-8B43.71 ± [0.51]1.91 ± [0.08]
+ DAPO50.22 ± [0.68]0.83 ± [0.05]
ReTool52.92 ± [0.25]9.26 ± [0.11]
GAPO-8B57.48 ± [0.30]0.37 ± [0.04]

2. Section 3.2 attributes GRPO to DeepSeek-R1 (2025), but it was introduced in DeepSeekMath (2024).

Thank you for pointing this out. We acknowledge this citation error and will correct the reference in Section 3.2 in the revised manuscript, citing DeepSeekMath (Shao et al., 2024) as the original work introducing GRPO.

AuthorsOpenReview ↗

Follow-Up on Responses

Dear Reviewer K5up,

As the discussion phase approaches its conclusion, we would like to respectfully inquire if there are any remaining concerns regarding our submission.

We are completely open to providing any further clarifications needed. If our responses have resolved your concerns, we would highly appreciate it if you could consider updating the final score.

Thank you very much for your time and response.

Best regards,

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their feedback and evaluation. The reviews raised several concerns regarding the novelty of our work, baseline comparisons, generalization, computational overhead, and experimental reliability. In response, we revised the manuscript and conducted additional experiments and analyses. The main revisions are summarized below.

(1) Novelty and related work. We expanded the Related Work section to include recent approaches such as SMART, Tool-Light, EAPO, AutoSearch, SAAS, IGPO, and TIPS. We clarified that our contribution lies not in the first identification of tool overuse, but in quantifying the marginal correctness gain of individual tool calls and incorporating it into policy optimization. Specifically, GAPO uses paired early-stopping branches to estimate the observable correctness gain of each tool call. Unlike approaches that primarily rely on changes in answer likelihood, GAPO explicitly identifies and penalizes outcome-redundant calls, thereby discouraging unnecessary tool interactions while preserving beneficial ones.

(2) Additional baselines and controlled comparisons. We added comparisons with DAPO (No-Tool), DAPO (Reward Shaping), and EAPO. We also extended the same-reward DAPO comparisons to Qwen3-4B and Qwen3-32B. Across Qwen3-4B/8B/32B, GAPO improves average accuracy from 48.45%/50.22%/52.09% to 52.98%/57.48%/59.43%, while reducing average tool calls from 0.24/0.83/1.89 to 0.20/0.37/0.77, respectively.

(3) Generalization to agentic search. We added experiments on HotpotQA, 2WikiMultiHopQA, and Bamboogle. Comparisons with EAPO and AutoSearch on Qwen2.5 models, as well as TIR and DAPO on Qwen3 models, show that GAPO achieves competitive or superior answer quality with fewer tool calls. These results further demonstrate the generalization of GAPO beyond mathematical reasoning and code generation.

(4) Computational overhead and efficiency. We added a detailed training overhead analysis in Appendix H. GAPO introduces 1.29×–1.54× per-step training time compared with DAPO, while maintaining comparable peak memory utilization. Its additional branch-generation complexity scales linearly with the number of tool calls, i.e., O(M(N+1)). We also report that GAPO reduces the average generated tokens per trajectory to 8.16K, compared with 11.76K for DAPO and 13.52K for ReTool.

(5) Methodological clarifications and statistical analysis. We clarified the estimation of minimum tool calls, including the treatment of trajectories without correct early-stopping branches. We also added mean and standard deviation results across eight independent evaluation runs in Appendix K and corrected the original GRPO citation in Section 2.2.

Summary. Overall, these revisions address the major concerns raised by the reviewers through additional experiments, methodological clarifications, and expanded analyses. The results further support our central finding that fine-grained tool-call supervision enables more effective tool use while maintaining or improving reasoning performance. We sincerely thank the reviewers again for their valuable suggestions, which have helped improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.