- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the tool-use collapse and excessive tool calling in small multimodal agents and proposes Agentic Necessity-Aware Policy Optimization (ANPO) . The method assigns separate learning signals to decision, execution, and answer stages, and judges the tool-call decision by the necessity of the tool for each question rather than by the outcome of each trajectory. Experiments use Qwen3.5-2B and 4B with four tools, reporting higher average accuracy and more selective tool use than the evaluated Tool-RLVR methods in the fifteen benchmarks covering the recognition, reasoning and perception tasks.
Critical strengths
- The paper addresses an important problem with targeted diagnostic evidence. Small agents must learn both how to use tools and when to invoke them, whereas final-answer rewards may not distinguish these capabilities effectively.
- The main evaluation is extensive covering several important experimental factors. Table 3 covers fifteen benchmarks spanning recognition, reasoning, and perception. ANPO achieves the highest accuracies compared with other methods.
Critical weaknesses
- The incremental gains over the strongest baseline lack training replication and uncertainty estimates. As shown in table 3, ANPO improves average accuracy over AXPO by only 1.4 and 0.8 percentage points at 2B and 4B, respectively. It is difficult to judge whether these gains are robust to training and evaluation randomness.
- For the experiment every method samples eight rollouts with four tool-enabled and four tool-disabled rollouts, and the tool-enabled success rate is computed only over trajectories that actually invoke a tool. The paper does not discuss how sensitive ANPO is to this effective sample size.
Questions for authors
- Do ANPO’s accuracy gains over AXPO remain consistent across different training and inference settings,such as the seeds and decoding configurations? Could the authors quantify the uncertainty in these improvements by reporting variability across runs and confidence intervals for the performance differences for the benchmark in table 3?
- How sensitive are the results to the number of actual tool-using trajectories, the allocation of rollouts between arms, and the decision threshold?
3 comments
Author Response (2/2)
W2 (Q2). Sensitivity of ANPO to the effective sample size of the tool-on estimate.
Thank you for raising this important point. We agree that the sensitivity to the effective sample size should be examined, and we analyzed it from two aspects: 1) the number of rollouts allocated to each arm and 2) the uncertainty of the tool-on estimate.
1) Regarding the number of rollouts,
We trained ANPO with different allocations of the eight rollouts between the tool-on and tool-off arms (: = 2:6 and 6:2), as well as with larger budgets (8:4 and 4:8). All other settings are identical to those of the main experiments. In particular, the 2:6 setting directly reduces the number of tool-using trajectories available for the tool-on estimate.
| Budget | MathVerse | MathVista | MathVision | DynaMath | LogicVista | Avg | ||
|---|---|---|---|---|---|---|---|---|
| Equal | 4 | 4 | 67.7 | 78.0 | 44.6 | 69.3 | 58.0 | 63.5 |
| 2 | 6 | 67.3 | 78.0 | 45.0 | 69.4 | 58.3 | 63.6 | |
| 6 | 2 | 66.7 | 77.9 | 43.5 | 69.1 | 57.4 | 62.9 | |
| Additional | 8 | 4 | 67.7 | 77.9 | 45.2 | 69.3 | 58.3 | 63.7 |
| 4 | 8 | 68.2 | 78.1 | 46.1 | 69.5 | 57.9 | 64.0 |
| Budget | V$^{\star}$ | HR-Bench | TallyQA | VisualProbe$_{\text{H}}$ | VLMsAreBlind | BabyVision | Avg | ||
|---|---|---|---|---|---|---|---|---|---|
| Equal | 4 | 4 | 82.9 | 75.7 | 84.0 | 23.6 | 66.3 | 15.4 | 58.0 |
| 2 | 6 | 82.7 | 74.3 | 82.7 | 23.6 | 64.9 | 15.5 | 57.3 | |
| 6 | 2 | 84.5 | 74.7 | 83.6 | 22.9 | 65.9 | 16.1 | 58.0 | |
| Additional | 8 | 4 | 86.4 | 75.4 | 84.7 | 25.2 | 66.8 | 16.2 | 59.1 |
| 4 | 8 | 84.0 | 74.9 | 82.3 | 24.3 | 66.2 | 15.9 | 57.9 |
Table. Pass@1 (%) under different rollout allocations on the reasoning (top) and perception (bottom) benchmarks. The 4:4 allocation is the setting used in the paper.
We can observe the following: 1) Under the equal budget, ANPO performs stably across allocations: the overall average accuracy (total 11 benchmarks) ranges from 60.2 to 60.5, staying within 0.4 points of the 4:4 allocation and above AXPO (59.0), the strongest baseline in Table 3, in every allocation. Even with only two tool-on rollouts (2:6), ANPO achieves 60.2 on average. 2) Under both the equal and additional budgets, allocating more rollouts to the tool-on arm favors perception, while allocating more rollouts to the tool-off arm favors reasoning. This is likely because the arm with more rollouts provides more learning signal for its own mode. Considering 1) and 2), ANPO is not sensitive to the effective sample size of the tool-on estimate.
2) Regarding the uncertainty of the tool-on estimate,
We measure how consistently the necessity decision () is estimated as the number of tool-on rollouts varies, using the agreement rate and Cohen's κ [1].
| 2 | 4 | 6 | 8 | 16 | 32 | |
|---|---|---|---|---|---|---|
| Agreement | 69.6 | 73.7 | 75.3 | 77.6 | 83.1 | 88.1 |
| Cohen's κ | 0.42 | 0.56 | 0.61 | 0.65 | 0.72 | 0.76 |
Table. Agreement (%) and Cohen's κ between the necessity decision estimated with tool-on rollouts and a reference decision estimated from 32 independently sampled tool-on rollouts.
The agreement increases with more tool-on rollouts, but remains close between 4 and 8 rollouts (κ = 0.56–0.65). Notably, even two independent estimates with 32 rollouts, eight times our budget, agree only at κ = 0.76, showing that some randomness from rollout sampling inevitably remains. Compared with this, four rollouts already achieve moderate agreement (κ = 0.56) according to the standard interpretation [2]. This indicates that the tool-on estimate with four rollouts is reasonably stable. Moreover, as shown in 1), the final performance of ANPO is maintained across different rollout allocations, suggesting that the remaining randomness in the tool-on estimate does not substantially affect training.
In addition, we checked the number of tool-using trajectories among the four tool-on rollouts on the perception questions of the training data. The fraction of questions with no tool call decreases from 19.5% to 17.1% and 9.6% over training, and the fraction with at most one tool call decreases from 34.3% to 28.4% and 23.0%. Thus, more than half of the questions, rising to 77.0% by the end of training, consistently have at least two tool-using trajectories for the tool-on estimate.
We added these analyses to App. D.7-D.8, where the revisions are highlighted in blue. We sincerely thank the reviewer for this insightful comment, which helped us further verify the stability of ANPO.
[1] A Coefficient of Agreement for Nominal Scales, Cohen, Educational and Psychological Measurement, 1960
[2] The Measurement of Observer Agreement for Categorical Data, Landis and Koch, Biometrics, 1977
Gentle Reminder
Dear Reviewer 5coM,
We sincerely appreciate your time and helpful questions on our work. In our response, we evaluated ANPO and AXPO across three training seeds and three inference seeds, showing that ANPO consistently outperforms AXPO with confidence intervals well above zero. We also analyzed the sensitivity to the rollout allocation and the number of tool-using trajectories, showing that ANPO remains robust in every setting.
As the discussion period ends soon, we would kindly ask whether our responses have addressed your concerns. We would be happy to address any remaining questions.
Author Response (1/2)
We thank the reviewer for the thoughtful review and the helpful questions on the robustness of our results. We address them below with additional runs across training seeds, inference seeds, and rollout allocations.
W1 (Q1). Robustness of the improvements over AXPO to training and evaluation randomness.
Thank you for raising this important point. We agree that the improvements of ANPO should be examined under both training and evaluation randomness, and we conducted additional experiments for each. Due to the limited time, we conducted these experiments on the 2B model.
1) Regarding training randomness,
We trained ANPO, AXPO, and GRPO with two additional independent seeds, in addition to the run reported in the paper. All other settings are identical to those of the main experiments.
Table. Pass@1 (%) of each training seed on the reasoning (top) and perception (bottom) benchmarks, and mean ± standard deviation of the average over three seeds. The confidence interval of the difference is computed with Welch's t-test over the three seeds.
ANPO outperforms AXPO on both the reasoning (63.3 ± 0.20 vs. 61.7 ± 0.28) and perception benchmarks (58.0 ± 0.11 vs. 56.4 ± 0.18) in all three seeds. The 95% confidence interval of the difference (ANPO − AXPO) is [1.04, 2.17] on reasoning and [1.25, 1.98] on perception, both well above zero. This indicates that the improvement of ANPO is not attributable to training randomness.
2) Regarding evaluation randomness,
We evaluated ANPO and AXPO with three inference seeds, the run reported in the paper and two additional seeds, under the same evaluation setting as Table 3.
Table. Pass@1 (%) on the reasoning (top) and perception (bottom) benchmarks, reported as mean ± standard deviation over three inference seeds.
ANPO outperforms AXPO with every inference seed, on both the reasoning (63.5 / 63.6 / 62.8 vs. 62.0 / 61.6 / 62.3) and perception benchmarks (58.0 / 58.3 / 58.1 vs. 56.5 / 56.6 / 56.4), where the first value of each triple is the run reported in the paper. On average, ANPO achieves 63.3 ± 0.48 vs. 62.0 ± 0.34 on reasoning and 58.1 ± 0.17 vs. 56.5 ± 0.09 on perception. This indicates that the improvement of ANPO is not attributable to evaluation randomness.
We added these results to App. D.6. We sincerely thank the reviewer for this valuable suggestion, which helped us verify the robustness of our results.