acceptodds
Under review as a conference paper at ICLR 2027

Easy to Call, Hard to Use Effectively: Learning How and When to Use Visual Tools in Small MLLM Agents

Abstract

Multimodal large language models (MLLMs) trained with reinforcement learning with verifiable rewards on tool-using trajectories (Tool-RLVR) have become agents that call visual tools such as zooming and Python code. Small MLLM agents are in growing demand for on-device applications, but turning small MLLMs into such agents remains difficult, as they struggle to use tools effectively. They fail to turn tool results into correct answers and keep calling tools even when the question does not need them, revealing limitations in how and when to use tools. We show that these two limitations hinder Tool-RLVR from training small MLLMs into effective agents, for two reasons. First, Tool-RLVR penalizes a proper tool-call decision whenever a later stage of the trajectory fails, so small MLLM agents decide not to use tools. Second, it ignores whether the tool was necessary for the question, so the call rate moves as a whole rather than following necessity. To address them, we propose Agentic Necessity-Aware Policy Optimization (ANPO), which rewards the decision, execution, and answer stages separately and judges the tool-call decision by the necessity of the tool for each question rather than by the outcome of each trajectory. Extensive experiments on fifteen benchmarks show that ANPO achieves the highest accuracy among Tool-RLVR methods while calling tools mainly where they are necessary.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Agentic Reinforcement Learning, Visual Tools

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_5coMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the tool-use collapse and excessive tool calling in small multimodal agents and proposes Agentic Necessity-Aware Policy Optimization (ANPO) . The method assigns separate learning signals to decision, execution, and answer stages, and judges the tool-call decision by the necessity of the tool for each question rather than by the outcome of each trajectory. Experiments use Qwen3.5-2B and 4B with four tools, reporting higher average accuracy and more selective tool use than the evaluated Tool-RLVR methods in the fifteen benchmarks covering the recognition, reasoning and perception tasks.

Critical strengths
  1. The paper addresses an important problem with targeted diagnostic evidence. Small agents must learn both how to use tools and when to invoke them, whereas final-answer rewards may not distinguish these capabilities effectively.
  2. The main evaluation is extensive covering several important experimental factors. Table 3 covers fifteen benchmarks spanning recognition, reasoning, and perception. ANPO achieves the highest accuracies compared with other methods.
Critical weaknesses
  1. The incremental gains over the strongest baseline lack training replication and uncertainty estimates. As shown in table 3, ANPO improves average accuracy over AXPO by only 1.4 and 0.8 percentage points at 2B and 4B, respectively. It is difficult to judge whether these gains are robust to training and evaluation randomness.
  2. For the experiment every method samples eight rollouts with four tool-enabled and four tool-disabled rollouts, and the tool-enabled success rate is computed only over trajectories that actually invoke a tool. The paper does not discuss how sensitive ANPO is to this effective sample size.
Questions for authors
  1. Do ANPO’s accuracy gains over AXPO remain consistent across different training and inference settings,such as the seeds and decoding configurations? Could the authors quantify the uncertainty in these improvements by reporting variability across runs and confidence intervals for the performance differences for the benchmark in table 3?
  2. How sensitive are the results to the number of actual tool-using trajectories, the allocation of rollouts between arms, and the decision threshold?
3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer for the thoughtful review and the helpful questions on the robustness of our results. We address them below with additional runs across training seeds, inference seeds, and rollout allocations.

W1 (Q1). Robustness of the improvements over AXPO to training and evaluation randomness.

Thank you for raising this important point. We agree that the improvements of ANPO should be examined under both training and evaluation randomness, and we conducted additional experiments for each. Due to the limited time, we conducted these experiments on the 2B model.

1) Regarding training randomness,

We trained ANPO, AXPO, and GRPO with two additional independent seeds, in addition to the run reported in the paper. All other settings are identical to those of the main experiments.

MethodMathVerseMathVistaMathVisionDynaMathLogicVistaAvg
ANPO67.7 / 67.5 / 67.678.0 / 77.9 / 78.244.6 / 44.2 / 43.869.3 / 69.0 / 68.758.0 / 57.3 / 57.663.3 ± 0.20
AXPO66.4 / 66.2 / 66.976.5 / 76.0 / 76.742.6 / 41.6 / 41.068.6 / 68.7 / 67.955.9 / 55.4 / 54.961.7 ± 0.28
GRPO65.9 / 66.0 / 66.276.4 / 76.3 / 75.941.8 / 42.2 / 42.667.1 / 67.3 / 66.955.5 / 53.7 / 55.861.3 ± 0.19
MethodV$^{\star}$HR-BenchTallyQAVisualProbe$_{\text{H}}$VLMsAreBlindBabyVisionAvg
ANPO82.9 / 84.4 / 84.475.7 / 74.6 / 74.984.0 / 84.4 / 83.923.6 / 23.2 / 22.966.3 / 66.2 / 65.915.4 / 15.8 / 15.358.0 ± 0.11
AXPO79.8 / 80.6 / 79.374.8 / 74.1 / 74.381.8 / 82.2 / 81.922.8 / 22.6 / 22.564.9 / 64.8 / 65.014.8 / 14.5 / 14.056.4 ± 0.18
GRPO77.7 / 77.9 / 80.273.6 / 72.6 / 73.281.1 / 81.3 / 80.920.7 / 20.5 / 20.963.1 / 64.2 / 63.514.1 / 14.4 / 13.955.2 ± 0.20

Table. Pass@1 (%) of each training seed on the reasoning (top) and perception (bottom) benchmarks, and mean ± standard deviation of the average over three seeds. The confidence interval of the difference is computed with Welch's t-test over the three seeds.

ANPO outperforms AXPO on both the reasoning (63.3 ± 0.20 vs. 61.7 ± 0.28) and perception benchmarks (58.0 ± 0.11 vs. 56.4 ± 0.18) in all three seeds. The 95% confidence interval of the difference (ANPO − AXPO) is [1.04, 2.17] on reasoning and [1.25, 1.98] on perception, both well above zero. This indicates that the improvement of ANPO is not attributable to training randomness.

2) Regarding evaluation randomness,

We evaluated ANPO and AXPO with three inference seeds, the run reported in the paper and two additional seeds, under the same evaluation setting as Table 3.

MethodMathVerseMathVistaMathVisionDynaMathLogicVistaAvg
ANPO67.4 ± 0.6077.9 ± 0.4944.5 ± 0.4569.2 ± 0.8357.6 ± 0.3563.3 ± 0.48
AXPO66.7 ± 0.5276.7 ± 0.3242.2 ± 0.9468.6 ± 0.5355.7 ± 0.5262.0 ± 0.34
MethodV$^{\star}$HR-BenchTallyQAVisualProbe$_{\text{H}}$VLMsAreBlindBabyVisionAvg
ANPO84.0 ± 1.1675.1 ± 0.7983.3 ± 0.6523.2 ± 0.5067.1 ± 0.6716.3 ± 1.1058.1 ± 0.17
AXPO80.8 ± 0.9474.0 ± 0.8281.9 ± 0.3221.9 ± 0.8165.6 ± 0.6514.9 ± 0.1056.5 ± 0.09

Table. Pass@1 (%) on the reasoning (top) and perception (bottom) benchmarks, reported as mean ± standard deviation over three inference seeds.

ANPO outperforms AXPO with every inference seed, on both the reasoning (63.5 / 63.6 / 62.8 vs. 62.0 / 61.6 / 62.3) and perception benchmarks (58.0 / 58.3 / 58.1 vs. 56.5 / 56.6 / 56.4), where the first value of each triple is the run reported in the paper. On average, ANPO achieves 63.3 ± 0.48 vs. 62.0 ± 0.34 on reasoning and 58.1 ± 0.17 vs. 56.5 ± 0.09 on perception. This indicates that the improvement of ANPO is not attributable to evaluation randomness.

We added these results to App. D.6. We sincerely thank the reviewer for this valuable suggestion, which helped us verify the robustness of our results.

AuthorsOpenReview ↗

Author Response (2/2)

W2 (Q2). Sensitivity of ANPO to the effective sample size of the tool-on estimate.

Thank you for raising this important point. We agree that the sensitivity to the effective sample size should be examined, and we analyzed it from two aspects: 1) the number of rollouts allocated to each arm and 2) the uncertainty of the tool-on estimate.

1) Regarding the number of rollouts,

We trained ANPO with different allocations of the eight rollouts between the tool-on and tool-off arms (NTN_T:NRN_R = 2:6 and 6:2), as well as with larger budgets (8:4 and 4:8). All other settings are identical to those of the main experiments. In particular, the 2:6 setting directly reduces the number of tool-using trajectories available for the tool-on estimate.

BudgetNTN_TNRN_RMathVerseMathVistaMathVisionDynaMathLogicVistaAvg
Equal4467.778.044.669.358.063.5
2667.378.045.069.458.363.6
6266.777.943.569.157.462.9
Additional8467.777.945.269.358.363.7
4868.278.146.169.557.964.0
BudgetNTN_TNRN_RV$^{\star}$HR-BenchTallyQAVisualProbe$_{\text{H}}$VLMsAreBlindBabyVisionAvg
Equal4482.975.784.023.666.315.458.0
2682.774.382.723.664.915.557.3
6284.574.783.622.965.916.158.0
Additional8486.475.484.725.266.816.259.1
4884.074.982.324.366.215.957.9

Table. Pass@1 (%) under different rollout allocations on the reasoning (top) and perception (bottom) benchmarks. The 4:4 allocation is the setting used in the paper.

We can observe the following: 1) Under the equal budget, ANPO performs stably across allocations: the overall average accuracy (total 11 benchmarks) ranges from 60.2 to 60.5, staying within 0.4 points of the 4:4 allocation and above AXPO (59.0), the strongest baseline in Table 3, in every allocation. Even with only two tool-on rollouts (2:6), ANPO achieves 60.2 on average. 2) Under both the equal and additional budgets, allocating more rollouts to the tool-on arm favors perception, while allocating more rollouts to the tool-off arm favors reasoning. This is likely because the arm with more rollouts provides more learning signal for its own mode. Considering 1) and 2), ANPO is not sensitive to the effective sample size of the tool-on estimate.

2) Regarding the uncertainty of the tool-on estimate,

We measure how consistently the necessity decision (rˉT>rˉR\bar{r}_T>\bar{r}_R) is estimated as the number of tool-on rollouts varies, using the agreement rate and Cohen's κ [1].

NTN_T24681632
Agreement69.673.775.377.683.188.1
Cohen's κ0.420.560.610.650.720.76

Table. Agreement (%) and Cohen's κ between the necessity decision estimated with NTN_T tool-on rollouts and a reference decision estimated from 32 independently sampled tool-on rollouts.

The agreement increases with more tool-on rollouts, but remains close between 4 and 8 rollouts (κ = 0.56–0.65). Notably, even two independent estimates with 32 rollouts, eight times our budget, agree only at κ = 0.76, showing that some randomness from rollout sampling inevitably remains. Compared with this, four rollouts already achieve moderate agreement (κ = 0.56) according to the standard interpretation [2]. This indicates that the tool-on estimate with four rollouts is reasonably stable. Moreover, as shown in 1), the final performance of ANPO is maintained across different rollout allocations, suggesting that the remaining randomness in the tool-on estimate does not substantially affect training.

In addition, we checked the number of tool-using trajectories among the four tool-on rollouts on the perception questions of the training data. The fraction of questions with no tool call decreases from 19.5% to 17.1% and 9.6% over training, and the fraction with at most one tool call decreases from 34.3% to 28.4% and 23.0%. Thus, more than half of the questions, rising to 77.0% by the end of training, consistently have at least two tool-using trajectories for the tool-on estimate.

We added these analyses to App. D.7-D.8, where the revisions are highlighted in blue. We sincerely thank the reviewer for this insightful comment, which helped us further verify the stability of ANPO.

[1] A Coefficient of Agreement for Nominal Scales, Cohen, Educational and Psychological Measurement, 1960

[2] The Measurement of Observer Agreement for Categorical Data, Landis and Koch, Biometrics, 1977

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 5coM,

We sincerely appreciate your time and helpful questions on our work. In our response, we evaluated ANPO and AXPO across three training seeds and three inference seeds, showing that ANPO consistently outperforms AXPO with confidence intervals well above zero. We also analyzed the sensitivity to the rollout allocation and the number of tool-using trajectories, showing that ANPO remains robust in every setting.

As the discussion period ends soon, we would kindly ask whether our responses have addressed your concerns. We would be happy to address any remaining questions.

Reviewer_5yWcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why small multimodal language models struggle to learn effective visual-tool use with Tool-RLVR. It identifies two issues: trajectory-level rewards can penalize a proper tool-call decision when a later stage fails, and existing methods optimize tool use without accounting for whether the tool is necessary for each question. To address them, the paper proposes Agentic Necessity-Aware Policy Optimization (ANPO), which assigns separate advantages to the decision, execution, and answer stages and estimates tool necessity online by comparing tool-on and tool-off rollouts. Experiments with Qwen3.5-2B and 4B across fifteen benchmarks show that ANPO improves accuracy while using tools more selectively than existing Tool-RLVR methods.

Critical strengths
  • The paper provides a clear and well-motivated analysis of tool collapse in small MLLM agents, distinguishing failures in deciding, executing, and answering rather than treating tool use as a single action.
  • ANPO is simple and introduces no additional rollouts relative to standard training. The experiments are extensive, covering two model sizes, fifteen benchmarks, several strong baselines, detailed ablations, and analyses of tool-call rate, difficulty adaptation, and redundant calls.
Critical weaknesses
  1. SRA does not fully separate the tool-call decision from downstream outcomes.

    • The decision reward is still computed from final-answer correctness. A proper tool-call decision may therefore receive a negative update when the model fails to interpret the tool result or produce the correct answer. Table 1 also shows that the broadcast reward leaves only 42.2% of proper decisions unpenalized. SRA thus reduces, rather than eliminates, the effect of downstream failures on decision learning.
  2. The two system prompts differ in reasoning guidance beyond what is necessary to enable or disable tools.

    • Although tool-on and tool-off prompts must necessarily differ, the tool-on prompt additionally instructs the model to assess tool necessity, reflect on tool results, and stop once sufficient information is obtained, whereas the tool-off prompt only requests step-by-step reasoning. These extra constraints may independently affect answer quality, so r_T - r_R may capture both the benefit of tool use and the benefit of additional reasoning guidance. A prompt-matched control that minimizes nonessential differences would make the claimed tool benefit easier to interpret.
Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank the reviewer for the careful review. The two points concern the scope of SRA and the design of the system prompts, and we address them below with additional analyses and a prompt-matched control.

W1. SRA does not fully separate the tool-call decision from downstream outcomes.

Thank you for raising this important point. We agree that SRA reduces, rather than eliminates, the effect of downstream failures on decision learning, and we revised the paper accordingly.

Since the decision reward of SRA is computed from the final-answer correctness of the tool-using trajectories, a proper tool-call decision could receive a negative advantage when the model fails to interpret the tool results or to produce the correct answer from them. We therefore revised the description of SRA in Sec. 1 and Sec. 4 so that it is no longer presented as fully separating the decision from downstream correctness. We also added this point to the Limitations section (App. F).

We would also like to respectfully clarify that the goal of SRA was to minimize the fraction of proper tool-call decisions penalized by final-answer failures, with only a minimal change in reward design and without any additional rollouts. As shown in Sec. 3.1, under trajectory-level rewards, the failure of a single trajectory directly suppresses the proper tool-call decision. SRA therefore shifts the decision reward from the trajectory level to the question level, so that a single failure has less influence on the decision. Accordingly, Table 1 was intended to show that SRA recovers the most proper decisions (42.2%) among reward designs that use only the information available in the training rollouts.

To examine how this fraction changes during training, we tracked it throughout Tool-RLVR training for GRPO, AXPO, and ANPO. The fraction is computed in the same way as in Table 1, over the proper tool-call decisions in failed tool-using trajectories of each method.

Method1-30 Steps31-60 Steps61-90 Steps91-120 Steps
GRPO0.00.00.00.0
AXPO39.443.746.844.4
ANPO36.744.654.358.7

Table. Fraction (%) of proper tool-call decisions left unpenalized in failed trajectories during Tool-RLVR training, computed as in Table 1 and averaged over each 30-step window (higher is better).

Under ANPO, the fraction rises steadily from 36.7% in the first 30 steps to 58.7% in the last 30 steps. In contrast, AXPO does not show a consistent increase, staying between 39.4% and 46.8%. GRPO remains at 0.0%, since every failed trajectory directly receives a negative advantage. Thus, although ANPO does not remove the misaligned decision credit at once, it progressively reduces it as training proceeds.

We added this analysis to App. A.4, and we sincerely thank the reviewer for this insightful comment, which helped us clarify the scope of SRA.

AuthorsOpenReview ↗

Author Response (2/2)

W2. The two system prompts differ in reasoning guidance beyond what is necessary to enable or disable tools.

Thank you for this thoughtful suggestion. We agree that a prompt-matched control helps to interpret rˉT−rˉR\bar{r}_T - \bar{r}_R, and we conducted this control in the following experiments.

We consider three prompt settings that differ only in the system prompts:

(1) Original: the default prompts used in the paper.

(2) Simple Tool-On: the guidance is removed from the tool-on prompt, leaving only the role, answer format, and tool descriptions, while the tool-off prompt is unchanged.

(3) Guided Tool-Off: tool-independent guidance similar to that of the tool-on prompt is added to the tool-off prompt, while the tool-on prompt is unchanged.

Under these settings, we examine two aspects of our ANPO model: (i) the necessity decision on the training data and (ii) the tool-on / tool-off performance gap on the perception benchmarks in Figure 13. For (i), we newly sampled four tool-on and four tool-off rollouts per question under each setting, following the training configuration. For (ii), we followed the same inference setting as in the paper, with four rollouts per question.

(i) Necessity decision on the training data.

Original vs Simple Tool-OnOriginal vs Guided Tool-Off
Spearman of rˉT\bar{r}_T0.810.90
Spearman of rˉR\bar{r}_R0.910.76
Necessity agreement (%)80.078.9

Table. Spearman correlation of the per-question rˉT\bar{r}_T and rˉR\bar{r}_R, and agreement rate of the necessity decision, with respect to the Original setting on the training data.

The modified prompts preserve the necessity decision for 80.0% (Simple Tool-On) and 78.9% (Guided Tool-Off) of the questions. However, please kindly note that the necessity estimate itself involves some randomness from rollout sampling: even when the original prompts are sampled twice independently, the decision agrees for 86.0% of the questions. Likewise, the arm whose prompt is unchanged in each setting yields Spearman correlations of 0.91 (rˉR\bar{r}_R under Simple Tool-On) and 0.90 (rˉT\bar{r}_T under Guided Tool-Off), which reflect sampling randomness alone. The modified arms still yield 0.81 and 0.76. Compared with these levels, the change caused by the modified prompts is small.

(ii) Tool-on / tool-off performance gap in Figure 13.

OriginalSimple Tool-OnGuided Tool-Off
Gap of Tool-On/Off+2.42+2.71+2.52
TCR at Tool-On57.5%59.4%57.9%

Table. Pass@1 difference between tool-on and tool-off (tool-on − tool-off), averaged over the six perception benchmarks in Figure 13, and the tool-call rate under the tool-on prompt.

The gap stays at +2.71 (Simple Tool-On) and +2.52 (Guided Tool-Off), close to +2.42 under the Original prompts, and the tool-call rate also changes little (57.5% vs 59.4% and 57.9%). Since the three settings differ only in the system prompts, this indicates that the additional guidance in the tool-on prompt has little effect on the measured benefit of tool use.

Lastly, we would like to respectfully note that such additional guidance in the tool-on prompt, such as assessing whether a tool is needed and reflecting on tool results, is commonly adopted in MLLM tool agents [1, 2, 3].

We added this analysis, together with the full prompts for all three settings, to App. D.4, where the revisions are highlighted in blue, and we sincerely thank the reviewer for this thoughtful suggestion.

[1] Agent Explorative Policy Optimization for Multimodal Agentic Reasoning, Kang et al., NeurIPS'26

[2] PyVision-RL: Forging Open Agentic Vision Models via RL, Zhao et al., ICML'26

[3] DeepEyesV2: Toward Agentic Multimodal Model, Hong et al., ICLR'26

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 5yWc,

We sincerely appreciate your time and thoughtful feedback on our work. In our response, we clarified the scope and motivation of SRA, showing through an additional experiment that ANPO progressively reduces the penalty on proper tool-call decisions during training. We also conducted the prompt-matched control you suggested, which showed that the additional guidance in the tool-on prompt has little effect on the measured benefit of tool use.

As the discussion period ends soon, we would kindly ask whether our responses have addressed your concerns. We would be happy to address any remaining questions.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Author Response (1/3)

We thank the reviewer for the constructive suggestions. We address each point below with a controlled comparison against necessity-aware methods and additional analyses on the stability of the necessity estimate.

W1 (Q1). Comparison with the closest necessity-aware tool-use methods.

Thank you for this valuable suggestion. We agree that a direct comparison with necessity-aware methods is important to support our contribution, and we conducted this comparison.

Following the reviewer's suggestion, we additionally compared ANPO with Look-Less [1] and AdaTooler-V [2], reimplemented in our framework. All methods start from the same SFT model of Qwen3.5-2B backbone and use the same training data, tools, and rollout budget, differing only in how tool necessity is incorporated into training. We compare these methods from two perspectives: 1) performance and 2) tool-use behavior.

1) Regarding performance,

In the tables below, we compare both methods with ANPO and AXPO, the strongest baseline in Table 3.

MethodMathVerseMathVistaMathVisionDynaMathLogicVistaAvg
ANPO67.778.044.669.358.063.5
AXPO66.476.542.668.655.962.0
Look-Less66.276.340.567.855.061.2
AdaTooler-V66.177.141.767.954.961.5
MethodV$^{\star}$HR-BenchTallyQAVisualProbe$_{\text{H}}$VLMsAreBlindBabyVisionAvg
ANPO82.975.784.023.666.315.458.0
AXPO79.874.881.822.864.914.856.5
Look-Less81.273.981.621.762.514.155.8
AdaTooler-V81.973.581.220.962.914.655.9

Table. Pass@1 (%) of necessity-aware methods on the reasoning (top) and perception (bottom) benchmarks, under the same evaluation setting as Table 3.

Both methods achieve lower accuracy than ANPO on average, on both the reasoning benchmarks (61.2 and 61.5 vs. 63.5) and the perception benchmarks (55.8 and 55.9 vs. 58.0). They even fall below AXPO (62.0 and 56.5). This shows that ANPO outperforms not only general Tool-RLVR methods but also existing necessity-aware methods.

2) Regarding tool-use behavior,

In the table below, we report the tool-call rate (TCR) and the redundant call rate of each method.

MethodReasoning TCRPerception TCRRedundant Tool Call (↓)
ANPO7.757.55.1
AXPO36.178.816.8
Look-Less27.667.326.2
AdaTooler-V1.823.914.7

Table. Tool-call rate (TCR, %) and redundant call rate (%) of necessity-aware methods, measured in the same way as in the paper.

Although both methods are trained to consider tool necessity, they still fall into the two failure modes discussed in Figure 1. Look-Less shows a high TCR (27.6% / 67.3% on reasoning / perception) with the highest redundant call rate (26.2%), indicating tool overuse. In contrast, AdaTooler-V rarely calls the tool even on the perception benchmarks (23.9%), similar to the tool underuse of GRPO. ANPO achieves the lowest redundant call rate (5.1%).

Considering 1) and 2) together, ANPO not only achieves better performance but also shows more effective tool use than existing necessity-aware methods.

Additionally, we would like to kindly note that AutoTool [3] is not included in this controlled comparison. AutoTool requires the model to explicitly select between tool-on and tool-off modes through special tokens. Since our SFT model is not trained with these additional tokens, applying AutoTool would require constructing new SFT data that contains the mode tokens and retraining the SFT model. This would break the identical starting point shared by all methods in our comparison.

We added these comparisons, together with the implementation details of each method and how it was adapted to our framework, to App. D.5. We sincerely thank the reviewer for this helpful suggestion.

[1] Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning, Li et al., ACL’26

[2] AdaTooler-V: Adaptive Tool-Use for Images and Videos, Wang et al., ACL Findings’26

[3] Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning, Ma et al., ICML’26

AuthorsOpenReview ↗

Author Response (2/3)

W2 (Q2). Stability of the necessity estimate. (1/2)

Thank you for raising this important point. We agree that the stability of the necessity estimate should be examined, and we analyzed it from three aspects: 1) sensitivity to the number of rollouts, 2) uncertainty of the necessity estimate, and 3) results across independent training seeds.

1) Regarding the sensitivity to the number of rollouts,

We trained ANPO with different allocations of the 8 rollouts between the tool-on and tool-off arms (NTN_T:NRN_R = 2:6 and 6:2), as well as with larger budgets (8:4 and 4:8). All other settings are identical to those of the main experiments.

BudgetNTN_TNRN_RMathVerseMathVistaMathVisionDynaMathLogicVistaAvg
Equal4467.778.044.669.358.063.5
2667.378.045.069.458.363.6
6266.777.943.569.157.462.9
Additional8467.777.945.269.358.363.7
4868.278.146.169.557.964.0
BudgetNTN_TNRN_RV$^{\star}$HR-BenchTallyQAVisualProbe$_{\text{H}}$VLMsAreBlindBabyVisionAvg
Equal4482.975.784.023.666.315.458.0
2682.774.382.723.664.915.557.3
6284.574.783.622.965.916.158.0
Additional8486.475.484.725.266.816.259.1
4884.074.982.324.366.215.957.9

Table. Pass@1 (%) under different rollout allocations on the reasoning (top) and perception (bottom) benchmarks. The 4:4 allocation is the setting used in the paper.

We can observe the following: 1) Under the equal budget, ANPO performs stably across allocations: the overall average accuracy (total 11 benchmarks) ranges from 60.2 to 60.5, staying within 0.4 points of the 4:4 allocation and above AXPO (59.0) in every allocation. The balanced allocation (4:4) achieves the best overall performance. 2) Under both the equal and additional budgets, allocating more rollouts to the tool-on arm favors perception, while allocating more rollouts to the tool-off arm favors reasoning. This is likely because the arm with more rollouts provides more learning signal for its own mode. Considering 1) and 2), ANPO is not sensitive to the number of rollouts in each arm.

2) Regarding the uncertainty of the necessity estimate,

We measure how consistently the necessity decision (rˉT>rˉR\bar{r}_T > \bar{r}_R) is estimated as the number of tool-on rollouts varies, using the agreement rate and Cohen's κ [1].

NTN_T24681632
Agreement69.673.775.377.683.188.1
Cohen's κ0.420.560.610.650.720.76

Table. Agreement (%) and Cohen's κ between the necessity decision estimated with NTN_T tool-on rollouts and a reference decision estimated from 32 independently sampled tool-on rollouts.

The agreement increases with more tool-on rollouts, but remains close between 4 and 8 rollouts (κ = 0.56–0.65). Notably, even two independent estimates with 32 rollouts, eight times our budget, agree only at κ = 0.76, showing that some randomness from rollout sampling inevitably remains. Compared with this, 4 rollouts already achieve moderate agreement (κ = 0.56) according to the standard interpretation [2]. This indicates that the necessity estimate with 4 rollouts is reasonably stable. Moreover, as shown in 1) and 3), the final performance of ANPO is maintained across different rollout allocations and training seeds, suggesting that the remaining randomness in the necessity estimate does not substantially affect training.

In addition, we checked the number of tool-using trajectories among the four tool-on rollouts on the perception questions of the training data. The fraction of questions with no tool call decreases from 19.5% to 9.6% over training, and more than half of the questions consistently have at least two tool-using trajectories, so tool calls are not rare in the tool-on arm.

[1] A Coefficient of Agreement for Nominal Scales, Cohen, Educational and Psychological Measurement, 1960

[2] The Measurement of Observer Agreement for Categorical Data, Landis and Koch, Biometrics, 1977

AuthorsOpenReview ↗

Author Response (3/3)

W2 (Q2). Stability of the necessity estimate. (2/2)

3) Regarding the results across independent training seeds,

We trained ANPO, AXPO, and GRPO with two additional independent seeds, in addition to the run reported in the paper.

MethodMathVerseMathVistaMathVisionDynaMathLogicVistaAvg
ANPO67.7 / 67.5 / 67.678.0 / 77.9 / 78.244.6 / 44.2 / 43.869.3 / 69.0 / 68.758.0 / 57.3 / 57.663.3 ± 0.20
AXPO66.4 / 66.2 / 66.976.5 / 76.0 / 76.742.6 / 41.6 / 41.068.6 / 68.7 / 67.955.9 / 55.4 / 54.961.7 ± 0.28
GRPO65.9 / 66.0 / 66.276.4 / 76.3 / 75.941.8 / 42.2 / 42.667.1 / 67.3 / 66.955.5 / 53.7 / 55.861.3 ± 0.19
MethodV$^{\star}$HR-BenchTallyQAVisualProbe$_{\text{H}}$VLMsAreBlindBabyVisionAvg
ANPO82.9 / 84.4 / 84.475.7 / 74.6 / 74.984.0 / 84.4 / 83.923.6 / 23.2 / 22.966.3 / 66.2 / 65.915.4 / 15.8 / 15.358.0 ± 0.11
AXPO79.8 / 80.6 / 79.374.8 / 74.1 / 74.381.8 / 82.2 / 81.922.8 / 22.6 / 22.564.9 / 64.8 / 65.014.8 / 14.5 / 14.056.4 ± 0.18
GRPO77.7 / 77.9 / 80.273.6 / 72.6 / 73.281.1 / 81.3 / 80.920.7 / 20.5 / 20.963.1 / 64.2 / 63.514.1 / 14.4 / 13.955.2 ± 0.20

Table. Pass@1 (%) of each training seed on the reasoning (top) and perception (bottom) benchmarks, and mean ± standard deviation of the average over three seeds. The confidence interval of the difference is computed with Welch's t-test over the three seeds.

ANPO outperforms AXPO on both the reasoning (63.3 ± 0.20 vs. 61.7 ± 0.28) and perception benchmarks (58.0 ± 0.11 vs. 56.4 ± 0.18) in all three seeds. The gap of 1.6 points on both groups is much larger than the standard deviation across seeds (at most 0.28). The 95% confidence interval of the difference (ANPO − AXPO) is [1.04, 2.17] on reasoning and [1.25, 1.98] on perception, both well above zero.

We added these analyses to App. D.6-D.8, where the revisions are highlighted in blue. We sincerely thank the reviewer for this valuable suggestion, which helped us further verify the stability of ANPO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.