acceptodds
Under review as a conference paper at ICLR 2027

Learn How Before When: Decoupling Capability and Routing for Visual Tool Use

29 views

Abstract

Tool-augmented reasoning requires learning to answer in both the direct and tool modes and to route each question appropriately. Existing reinforcement learning with verifiable rewards (RLVR) methods often train mode capability and routing jointly and can exhibit phenomena like tool overuse or tool collapse, yet the connection between these behaviors and joint training remains unclear. We analyze GRPO advantages and their effects on one-step changes in mode capability and routing. The analysis shows how a mode can lose sampling probability while its capability is still improving, and why reward incentives can favor tool use without a corresponding accuracy benefit. Training diagnostics under representative reward settings further illustrate sharp declines in mode frequency and persistent tool calls under sustained balancing. Motivated by these findings, we propose a two-stage framework that separates mode capability learning from routing. We first use GRPO to train a shared expert under explicit mode instructions, then compare its performance across modes to guide an autonomous student through on-policy self-distillation. This design transfers mode capabilities and question-specific routing without additional routing rewards or large-scale supervised tool-use trajectories. Experiments show accuracy gains of up to 14.13 percentage points over Qwen2.5-VL-7B on high-resolution perception benchmarks. Distillation process further substantially reducing tool use rate, supporting more effective tool-use decisions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Tool Use, Reinforcement Learning, Mode Competition, Learning Dynamics, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AsHkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a two-stage framework that first trains direct answering and tool-use capabilities separately using Direct/Tool GRPO, and then learns when to use tools through OPD. The overall idea is clear and the results are promising, but the current experiments don't fully justify the need for decoupling or show whether the model actually learns better tool routing.

Critical strengths
  • Separating Capability Learning from Routing Learning through Direct/Tool GRPO and OPD is intuitive and well-motivated.

  • The method improves accuracy across multiple visual benchmarks while reducing the tool-call rate, showing benefits in both effectiveness and efficiency.

Critical weaknesses
  • Higher accuracy and fewer tool calls could simply mean the model has become stronger at solving tasks without tools. The paper doesn't clearly separate capability gains from routing improvements, so it's hard to conclude that the model has actually learned better tool routing.

  • Missing key baselines. I'd like to see two additional comparisons:

    • End-to-end Joint RL: Compare the full two-stage approach against joint RL under the same data, rollout budget, and training compute. Otherwise, it's hard to justify why decoupling is necessary.

    • Separate Experts + OPD: Train independent Direct-only and Tool-only experts, then merge them through OPD. This would help explain why Stage I needs a shared dual-mode expert instead of two separate experts.

  • Stage II introduces several changes at once, making it hard to tell where the gains come from. I think some important baselines are missing, such as off-policy distillation with SFT, random mode assignment, and always-direct/always-tool teachers.

  • Qwen2.5-VL-7B doesn't have crop-and-zoom tool-use prior, while many recent open-source VLMs (after Qwen3-VL) already do. It's unclear whether the proposed method would still be effective for models with more mature tool-use capabilities.

  • In Appendix I. The text mentions 150 training steps and reports results from steps 101–150, but Figure 5 only shows 50 steps. This is strange.

Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer AsHk (1/2)

We sincerely thank the reviewer for their constructive feedback.

Q1: Higher accuracy and fewer tool calls could simply mean the model has become stronger at solving tasks without tools. The paper doesn't clearly separate capability gains from routing improvements, so it's hard to conclude that the model has actually learned better tool routing.

A1: We thank the reviewer for the suggestion.

To distinguish improved question-dependent routing from a simple preference for direct answers, we conduct an additional evaluation of the question-level optimal routing rate on HRBench. For each question, we identify the preferred mode by comparing the empirical accuracies of Direct and Tool, assigning ties to Direct to avoid unnecessary tool calls. We then measure how often the model's autonomous routing decision matches this preference. Stage II improves the optimal routing rate from 55.00% to 76.50% on HRBench-4K and from 52.63% to 73.25% on HRBench-8K, yielding gains of 21.50 and 20.63 percentage points, respectively. These substantial improvements directly demonstrate that Stage II learns to make more appropriate tool-use decisions for individual questions, rather than simply reducing tool usage.

This finding is further supported by Table 2 in our original paper, where our method consistently outperforms heuristic and learned routing baselines, including an LLM-based router, under comparable tool-use budgets. Together, these results provide consistent evidence that the improvements stem from more effective question-dependent routing, rather than solely from stronger direct-answering capabilities.

Q2: Missing key baselines. I'd like to see two additional comparisons:

  • End-to-end Joint RL: Compare the full two-stage approach against joint RL under the same data, rollout budget, and training compute. Otherwise, it's hard to justify why decoupling is necessary.
  • Separate Experts + OPD: Train independent Direct-only and Tool-only experts, then merge them through OPD. This would help explain why Stage I needs a shared dual-mode expert instead of two separate experts.

A2: Thanks for your suggestions.

We first clarify the motivations behind our two key design choices and then provide additional comparisons to substantiate their effectiveness.

Our primary motivation for decoupling is to avoid the training instability observed in joint RL, where mode collapse or tool overuse can cause optimization to concentrate on a single mode, limiting learning opportunities for the other. Regarding the motivation for using a shared expert, we do not claim it is the only viable approach to decoupling. Rather, it provides a more parameter-efficient solution by learning both mode capabilities within a single model, avoiding the additional training and model-management overhead of maintaining two independent experts during the distillation process.

Following your suggestions, we conduct additional experiments comparing our framework against (1) end-to-end joint RL under the same training budget with Mini-o3-like and DeepEyes rewards, and (2) independently trained Direct-only and Tool-only experts followed by on-policy distillation. The results are summarized below.

MethodV* BenchHRBench 4KHRBench 8KHRBench Avg.
End-to-end Joint RL (Mini-o3)85.3475.2569.3872.32
End-to-end Joint RL (DeepEyes)87.9674.5068.6371.57
Separate Experts + OPD86.9174.6367.0070.82
Ours90.0576.3871.2573.82

As shown in the table, our method consistently outperforms both compute-matched joint RL baselines, indicating that the improvements are not simply due to additional training computation. Furthermore, while separate experts provide an alternative for decoupling, our shared-expert approach achieves higher accuracy across all three benchmarks with a more parameter-efficient training design. These results further support both the effectiveness of decoupling capability learning from routing and the practical advantages of using a shared expert.

AuthorsOpenReview ↗

Response to Reviewer AsHk (2/2)

Q3: Stage II introduces several changes at once, making it hard to tell where the gains come from. I think some important baselines are missing, such as off-policy distillation with SFT, random mode assignment, and always-direct/always-tool teachers.

A3: Thanks for your suggestions.

Following your suggestions, we conduct three additional comparisons: (1) off-policy SFT using expert-generated trajectories, (2) on-policy distillation with random mode assignments, and (3) on-policy distillation with always-direct or always-tool teachers. The results are summarized below.

MethodV* BenchHRBench-4KHRBench-8KHRBench Avg.
SFT83.2574.2567.2570.75
OPD + Random Assignment89.0173.0068.8870.94
OPD + Always-Direct89.5374.8867.2571.07
OPD + Always-Tool89.5374.3868.7571.57
Ours90.0576.3871.2573.82

As shown in the table, our method consistently outperforms all ablation variants across the three benchmarks. Compared with off-policy SFT, our on-policy approach achieves substantial accuracy improvements, supporting the effectiveness of distillation on student-generated trajectories. Furthermore, Our method outperforms both random assignment and fixed-mode teachers, indicating the importance of selecting supervision based on the relative strengths of the two modes for each question. Together, these results support the effectiveness of combining question-specific mode assignment with on-policy self-distillation in Stage II.

Q4: Qwen2.5-VL-7B doesn't have crop-and-zoom tool-use prior, while many recent open-source VLMs (after Qwen3-VL) already do. It's unclear whether the proposed method would still be effective for models with more mature tool-use capabilities.

A4: Thanks for your suggestion.

We would like to emphasize that our method is motivated by the instability arising from jointly optimizing mode capabilities and routing, rather than limitations in the backbone's tool-use capabilities. While stronger tool-use priors may improve the model's ability to solve questions with tools, they do not inherently eliminate inter-mode competition during RL, which can still lead to unstable training behavior.

Following your suggestion, we conduct additional experiment to validate the effectiveness of our framework on Qwen3-VL-8B-Instruct,which have stronger tool-use priors. The results are summarized below.

ModelV* BenchHRBench 4KHRBench 8KHRBench Avg.
Qwen3-VL-8B Instruct87.43%76.38%69.12%72.75%
Stage I model88.48%79.50%70.12%74.81%
Stage II model90.58%81.00%73.38%77.19%

As shown in the table, stage II consistently improves upon Stage I across all benchmarks, with gains of 2.10 percentage points on VStar and 2.38 percentage points on HRBench overall. These results demonstrate that our two-stage framework remains effective on models with stronger tool-use priors, supporting its applicability beyond models with limited initial tool-use capabilities.

Q5: In Appendix I. The text mentions 150 training steps and reports results from steps 101–150, but Figure 5 only shows 50 steps. This is strange.

A5: Thank you for this feedback.

We apologize for the errors in Appendix I. The experiment was conducted for 50 training steps, as shown in Figure 5. The references to "150 training steps" and "steps 101–150" were typographical errors, which we have corrected in the revised manuscript.

Cool, I've carefully read the authors' response, and I'm overall satisfied with it. Most of my previous concerns have been addressed, so I've decided to increase my score.

Thank you for your encouraging feedback and for recognizing our efforts to address your concerns. We sincerely appreciate your thoughtful comments and constructive suggestions, which have helped us further strengthen our work.

Reviewer_9LzsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the instability of joint training for mode capability (direct answering vs. tool use) and routing in visual agents under RLVR. Through a question-level advantage decomposition and one-step dynamic analysis, the authors demonstrate that mode capability improvement does not necessarily lead to appropriate routing, often causing mode collapse or tool overuse. To address this, the authors propose a two-stage decoupled framework: Stage I trains a shared expert policy with forced mode instructions using within-group advantage normalization, and Stage II freezes the expert, evaluates its empirical accuracy per question to pick the preferred mode, and distills autonomous routing into a student model via on-policy self-distillation. Validated on high-resolution perception benchmarks and grounding tasks, the method improves accuracy while substantially reducing tool invocation frequency.

Critical strengths

The decomposition of GRPO advantages into inter-mode competition and within-mode learning provides a clean theoretical justification for why mode collapse occurs during joint RL training. The proposed two-stage decoupled pipeline achieves strong empirical accuracy gains while cutting tool-invocation rates significantly, leading to measurable inference speedups.

Critical weaknesses

The framework is evaluated only on a single tool type (cropping/zooming), leaving its generalizability to multi-tool scenarios unproven. While bypassing complex reward engineering, the need for separate Stage I training, offline trajectory evaluation across all training questions, and subsequent on-policy distillation increases pipeline complexity and data-budget demands.

Questions for authors

How does the Stage II mode selection behave when the performance gap between direct and tool modes is minimal, and did you evaluate sensitivity to the rollout budget k ? How easily does this decoupled formulation extend to multi-tool environments where routing involves a discrete choice among multiple heterogeneous tools rather than a binary decision?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9Lzs (1/2)

We sincerely thank the reviewer for recognizing the motivation behind our work and for their constructive feedback.

Q1: The framework is evaluated only on a single tool type (cropping/zooming), leaving its generalizability to multi-tool scenarios unproven.

A1: Thank you for your suggestion.

We would like to emphasize that our work primarily addresses whether to invoke tools, rather than which specific tool to select. Accordingly, our formulation distinguishes between direct and tool-assisted trajectories, regardless of the number or types of available tools. Therefore, our method, which separates learning tool-use capabilities from learning when to invoke tools, naturally extends to multi-tool settings.

If explicit routing among multiple heterogeneous tools is required, our framework can also be generalized to multiple modes. Stage I trains the capabilities of each mode separately, while Stage II identifies the preferred mode based on estimated success rates and distills these preferences into an autonomous routing policy. The main additional cost is the increased sampling required to train and evaluate multiple modes.

To further evaluate the generalizability of our framework beyond binary routing, we additionally extend our environment with an image rotation tool alongside cropping, enabling explicit three-way routing among Direct, Crop, and Rotate, and evaluate the proposed two-stage framework. The results are summarized below.

Table: Performance of multi-tool routing with heterogeneous visual tools.

ModelV* BenchHRBench 4KHRBench 8K
Qwen2.5-VL-7B Instruct75.92%68.88%64.50%
Stage I model91.10%74.63%66.13%
Stage II model91.62%74.75%68.00%

The results show that Stage II consistently improves upon Stage I on V* and HRBench, demonstrating the effectiveness of our two-stage framework in a multi-tool setting and further supporting its generalizability to multi-way routing scenarios.

Q2: While bypassing complex reward engineering, the need for separate Stage I training, offline trajectory evaluation across all training questions, and subsequent on-policy distillation increases pipeline complexity and data-budget demands.

A2: Thanks for your suggestion.

Regarding pipeline complexity and data budget requirements, we acknowledge that our framework introduces additional training steps compared with single-stage RL. However, the overall pipeline remains relatively straightforward: Stage I trains a single shared expert using mode-conditioned GRPO, followed by a one-time offline evaluation of mode preferences and an additional on-policy distillation stage. This design builds on established RL and distillation techniques without requiring elaborate reward engineering or separately trained expert models.

To quantify the computational overhead, we report the GPU-hours required by each component: 31.01 H200 GPU-hours for Stage I, 5.48 for offline mode assignment, and 16.18 for Stage II, totaling 52.67 H200 GPU-hours. For cross-hardware comparison, we normalize GPU-hours to A100 equivalents using a 2.56× throughput factor for H200, based on empirical training benchmarks [1]. We similarly normalize H100 GPU-hours using a 2.45× factor and approximate A800 GPU-hours as A100 equivalents.

Table: Comparison of Training Costs

MethodGPU TypeGPU-hoursEst. A100-equiv. GPU-hours
ERGOH100150.00~367.00
Pixel ReasonerA800640.00~640.00
OursH20052.67~134.84

As shown in the table, our complete pipeline requires approximately 134.84 A100-equivalent GPU-hours, representing 2.72× and 4.7× lower training compute than ERGO and Pixel Reasoner, respectively. These results demonstrate that our approach remains computationally efficient despite incorporating additional training stages, with substantially lower overall training costs than existing visual tool-use methods.

[1] Shad and Mazloumi. Which Kempner Partition Should Your Job Run On? Harvard Kempner Institute, 2026.

AuthorsOpenReview ↗

Response to Reviewer 9Lzs (2/2)

Q3: How does the Stage II mode selection behave when the performance gap between direct and tool modes is minimal, and did you evaluate sensitivity to the rollout budget k ?

A3: Thank you for your suggestions.

As acknowledged in our limitations, finite rollout budgets can introduce uncertainty into mode assignments, particularly when the performance gap between the two modes is small. However, the impact of such uncertainty on expected accuracy is naturally limited by the performance gap between the two modes.

To quantify the impact of incorrect mode selection, we consider the expected accuracy regret, defined as the performance loss relative to always selecting the better mode. Let pD(x)p_D(x) and pT(x)p_T(x) denote the true success probabilities of the two modes, and Perr(x;K)P_{\mathrm{err}}(x;K) the probability of selecting the inferior mode using KK independent rollouts per mode. The expected regret can be expressed as

RK(x)=∣pT(x)−pD(x)∣Perr(x;K).\mathcal R_K(x)=|p_T(x)-p_D(x)|P_{\mathrm{err}}(x;K).

This expression shows that the regret depends on both the performance gap and the probability of incorrect selection. When the two modes perform similarly, the performance gap is small, so even a relatively high probability of incorrect selection results in limited accuracy loss.

Additionally, to assess the sensitivity of mode assignments, we conduct additional experiments using the same frozen Stage-I checkpoint and 6,000 Stage-II training questions, comparing K = 4, 8, 16, and 32 samples per mode. We report two metrics: (1) Assignment disagreement, the fraction of questions whose assigned mode changes relative to K = 4, following our original assignment rule that favors Direct in case of ties; and (2) Strict reversal, the fraction of initially non-tied questions whose preference switches from Direct to Tool or vice versa.

Table: Sensitivity of Mode Preferences to Rollout Budget

Samples per modeAssignment disagreement (%) ↓Strict reversal (%) ↓
K = 4 → 86.803.78
K = 4 → 1610.639.05
K = 4 → 3213.8713.30

As shown in the table, assignment disagreement remains below 14% across all 6,000 questions, while strict preference reversals remain below 14% among initially non-tied questions, even when increasing K from 4 to 32. These results indicate that the majority of mode assignments remain consistent across sampling budgets.

Q4: How easily does this decoupled formulation extend to multi-tool environments where routing involves a discrete choice among multiple heterogeneous tools rather than a binary decision?

A4: Thank you for your question. We have answered this question in our response to Q1 and kindly refer the reviewer to A1 for further details.

AuthorsOpenReview ↗

Reminder of the Upcoming Rebuttal Discussion Deadline

Thank you again for your valuable feedback and constructive suggestions. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal and share your thoughts on our responses. We look forward to hearing your feedback.

AuthorsOpenReview ↗

Gentle Reminder: Final Day of the Discussion Period

Dear Reviewer,

As the discussion period is approaching its end, with only one day remaining, we would greatly appreciate hearing your feedback on our rebuttal and additional experiments.

We hope our responses have addressed your concerns, and we would welcome either a reconsideration of your rating or any remaining questions you may have. We are happy to provide further clarification if needed.

Thank you again for your time and consideration.

Reviewer_EQ3gOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to teach visual models both how to use tools and when to use them. The authors argue that learning these together can make the model favor one mode too early, leaving the other mode without enough training. They propose two stages. Stage I trains the same model with two instructions: answer directly or use the image-cropping tool, with GRPO advantages computed separately for each mode. Stage II compares the expert’s accuracy in both modes on each training question, then uses the preferred mode to guide an autonomous student through on-policy self-distillation. Results show improved accuracy on high-resolution visual benchmarks, while Stage II substantially reduces tool use compared with Stage-I autonomous inference.

Critical strengths
  1. I find the idea of learning both modes before learning when to use them interesting, although I am not sure how new Stage I is compared with previous work.
  2. Stage II is particularly interesting to me. Comparing the two modes on each training question gives the model a concrete signal about when tools are useful. The results also show that tool use can be reduced while improving accuracy.
Critical weaknesses

Stage I already gives most of the accuracy improvement on V*Bench, even in direct mode. However, there is no direct-only RL baseline where the model is only allowed to output direct answer or the model is only allowed to use tools. I am not sure which part of Stage I is responsible for this improvement.

Questions for authors
  1. How does the Stage-II model perform under forced direct and forced tool modes?
  2. With only four samples per mode, how stable are the mode assignments? How often do ties occur, and how much does the tie-breaking rule affect tool use?
3 comments
AuthorsOpenReview ↗

Response to Reviewer EQ3g (1/2)

We sincerely thank the reviewer for recognizing the motivation behind our work and for their constructive feedback.

Q1: Stage I already gives most of the accuracy improvement on V*Bench, even in direct mode. However, there is no direct-only RL baseline where the model is only allowed to output direct answer or the model is only allowed to use tools. I am not sure which part of Stage I is responsible for this improvement.

A1: Thank you for your suggestion.

The primary motivation behind Stage I stems from our analysis of inter-mode competition in joint RL. Therefore, our original comparisons focus on joint RL baselines to examine this issue. Training exclusively in either mode eliminates the inter-mode competition that Stage I is specifically designed to address, and thus cannot directly evaluate whether our method mitigates this issue.

Following your suggestion, we conduct additional experiments by training direct-only and tool-only policies under the same training budget and evaluating both policies under corresponding inference modes. The results are shown below.

Table:Performance of Single-Mode RL Training on V* and HRBench.

Training ModeV* BenchHRBench 4KHRBench 8K
Direct-only85.86%70.25%66.75%
Tool-only86.91%74.13%67.38%

As shown in the table, training either mode independently yields substantial improvements, but neither matches the performance of our dual-mode training. This further supporting our Stage I design, which explicitly preserves learning opportunities for both modes.

Q2: How does the Stage-II model perform under forced direct and forced tool modes?

A2: Thank you for your suggestion.

Stage I and Stage II serve different optimization objectives. While Stage I explicitly strengthens both direct and tool capabilities, Stage II focuses on learning an autonomous tool-use policy through on-policy self-distillation, rather than further optimizing either mode independently. Therefore, Stage II does not necessarily improve performance under forced execution modes, and minor fluctuations may occur.

Following the suggestion, we additionally evaluate the Stage-II model under forced direct and forced tool modes on HRBench.

Table: Stage-II Performance on HRBench under Forced Modes.

Execution Mode4K FSP4K FCP4K Overall8K FSP8K FCP8K Overall
Forced Direct87.0561.9874.5277.3357.7767.55
Forced Tool88.0561.7874.9177.4757.9167.69

Result shows that both forced modes exhibit modest performance variations, with slight improvements on HRBench-4K and slight decreases on HRBench-8K. These results are consistent with our objective of learning an effective autonomous tool-use policy rather than optimizing either forced mode independently.

AuthorsOpenReview ↗

Response to Reviewer EQ3g (2/2)

Q3: With only four samples per mode, how stable are the mode assignments? How often do ties occur, and how much does the tie-breaking rule affect tool use?

A3: Thank you for your suggestions.

To assess the stability of mode assignments, we conduct additional experiments using the same frozen Stage-I checkpoint and 6,000 Stage-II training questions, comparing K = 4, 8, 16, and 32 samples per mode. We report two metrics: (1) Assignment disagreement, the fraction of questions whose assigned mode changes relative to K = 4, following our original assignment rule that favors Direct in case of ties; and (2) Strict reversal, the fraction of initially non-tied questions whose preference switches from Direct to Tool or vice versa.

Table: Sensitivity of Mode Preferences to Rollout Budget

Samples per modeAssignment disagreement (%) ↓Strict reversal (%) ↓
K = 4 → 86.803.78
K = 4 → 1610.639.05
K = 4 → 3213.8713.30

As shown in the table, assignment disagreement remains below 14% across all 6,000 questions, while strict preference reversals remain below 14% among initially non-tied questions, even when increasing K from 4 to 32. These results indicate that the majority of mode assignments remain consistent across sampling budgets.

Regarding ties, among the 6,000 questions, 785 favor Direct, 696 favor Tool, and 4,264 (71.07%) exhibit equal non-zero correctness, while 255 yield zero correctness in both modes and are excluded. In principle, increasing the sampling budget allows finer-grained estimation of the performance difference between the two modes. However, such fine-grained distinctions are not always necessary for efficient tool use. Our goal is to identify instances where Tool offers a meaningful advantage, rather than resolve every marginal difference in conditional accuracy. In this sense, K=4K=4 provides a coarse-grained preference signal that prioritizes more pronounced performance differences. With this coarse-grained estimation, questions with small differences in conditional accuracy may receive the same estimated correctness, making tie-breaking an important part of routing supervision. We deliberately assign these ties to Direct, consistent with our efficiency objective of avoiding additional tool overhead when no clear performance benefit is observed.

AuthorsOpenReview ↗

Reminder of the Upcoming Rebuttal Discussion Deadline

Thank you again for your valuable feedback. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal and share your thoughts on our responses. We look forward to hearing your feedback.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. If you haven't already, we would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.

Thank you for your time and valuable contributions to the review process!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.