acceptodds
Under review as a conference paper at ICLR 2027

RL-Trained Crop-and-Zoom Is Often Decorative: Pixel Budgets, Protocol Costs, and Causal Credit

Abstract

We ask when crop-and-zoom tools trained with reinforcement learning (RL) help and why they often fail. Many tool-equipped vision-language models (VLMs) turn out to behave decoratively: they localize accurately, yet the returned crop adds little causal information. The cause is a pixel-budget gate: a perfect crop's value is set by the gap between an image's resolution and the model's working pixel budget, and in every configuration we measure grows strictly monotonically as that budget tightens (from +5.8 to +31.1 points across datasets and scales). Real above-budget imagery (MME-RealWorld-Lite, DocVQA) confirms the degradation the gate rests on: whole-image accuracy falls under a tight budget only for models that can use the pixels. Standard benchmarks and RL training pools (median 0.19M pixels) sit below that budget, so crops that are useless there carry recoverable information at deployment budgets. Where a crop does carry value, the standard additive tool-call serialization throws it away: a plain second image helps in all 16 Qwen dataset-by-budget comparisons (+6.8 to +17.3 points), but routing the identical pair through a tool-call exchange costs the untrained 3B base model 16–52 points across our budget sweeps, and larger untrained models 4–25 at 12.8M, by inducing tool-call loops; RL training only reduces that loss without learning to use the crop. Letting the returned crop replace the full image, or presenting it plainly, recovers most of the loss with no retraining. We audit tool use with counterfactual ablation credit: a call counts only if replacing its returned crop with a shifted crop flips the answer. On V* it is empirically nested within probe-style credit (p < 1e-4); yet, under the standard group relative policy optimization (GRPO) recipe, even content-causal credit with above-budget data does not yield durable, selective tool use. Together, these results reduce to three conditions governing when crop-and-zoom can pay: images served above the model's working budget, the crop presented in place of the full image, and content-causal credit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual tool use, reinforcement learning, reward design, counterfactual credit assignment, high-resolution perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_q1ZROpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when RL-trained crop-and-zoom tools improve VLM performance and why their use is often decorative.

The authors first identify a pixel-budget gate: the value of a correct crop is determined by the gap between an image’s native resolution and the model’s working pixel budget. Furthermore, by decomposing the tool-use protocol, they show that although presenting a crop improves performance, delivering the same crop through a standard tool-call exchange can introduce substantial performance losses. To measure whether a model actually relies on the returned visual content, the paper proposes counterfactual ablation credit, which replaces the crop with a shifted region and credits the call only when the answer changes. The authors find that most calls made by an RL-trained model are non-causal despite accurate localization. Finally, experiments with above-budget data and alternative advantage estimators show that causal rewards alone do not produce durable, selective tool use.

Critical strengths
  1. Important and well-supported findings. The paper challenges a common assumption that frequent or accurate tool calls necessarily indicate effective tool use. Through a carefully designed set of experiments, it examines the problem from several complementary perspectives, such as the pixel-budget gate, the cost of tool-call serialization, etc., and reaches the conclusion that current RL-trained crop-and-zoom behavior is largely decorative and becomes genuinely useful only under specific pixel-budget, presentation, and credit-assignment conditions.

  2. New insights for evaluating and improving tool use. The paper provides a useful perspective for studying whether tool calls really contribute to a model’s predictions, rather than merely correlating with successful outcomes. Its evaluation framework could be extended to other visual or multimodal tools, and motivate future work targeting more effective tool-using, such as tool-necessity training, better credit assignment, and more selective policies that invoke tools only when they provide additional information.

Critical weaknesses
  1. Limited generality of the training conclusions. Finding 3 focuses on the behavior of RL training, yet almost all training experiments are conducted on Qwen2.5-VL-3B. The current evidence is insufficient to establish that the observed collapse or decorative tool use generalizes broadly across different VLM backbones.

  2. Minor unclarity of the presentation. The paper contains many experiments, including the initial observation of decorative tool use and the three findings. Each experiment has substantially different protocols, datasets, metrics. Though the experiment setup is explained very clearly in the text, I would recommend that the paper can present a simple overview figure to show how the experiments are implemented and related, together with concrete case examples showing the complete pipeline of each experiment, which would make the methodology and conclusions much easier to follow.

Questions for authors
  1. (See Weakness 2) Could the authors add an overview diagram of the experimental pipeline or several concrete cases showing how each experiment is conducted instead of solely using textual explanations?

  2. The substantial performance degradation caused by the tool-call protocol in Finding 2 is somewhat counterintuitive. Could the authors provide a representative example, including the exact tool-call serialization, model outputs, and repeated-call behavior, to illustrate how this failure occurs?

  3. For locally-deployed small VLMs, it may not always be necessary to impose a tight working pixel budget. However, when using large proprietary models (e.g., GPT6-Astra), the number of visual tokens greatly affect the fee paid. I wonder whether the conclusions of Findings 1 and 2 also apply in this setting? I would be grateful if the authors could provide practical guidance on crop-and-zoom tool-using for large proprietary models (e.g., whether users should directly downsample full images or first crop them and use crop-only inputs to save token cost).

Reviewer_EHwkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates when RL-trained crop-and-zoom tools provide useful visual evidence. It varies working pixel budgets, compares image-presentation protocols, and measures answer dependence on returned crops through counterfactual replacement. It also incorporates this signal into RL rewards and studies training dynamics under GRPO, REINFORCE++, and RLOO. The authors report that tool calls can be largely decorative, that tool-mediated presentation can substantially degrade performance, and that counterfactual credit does not reliably produce persistent, selective tool use under the tested recipes.

Critical strengths
  1. The distinction between tool invocation and useful evidence acquisition is practically important. Comparing identical visual content across presentation conditions helps expose failures hidden by aggregate tool-available accuracy. The observation that persistent calling and content-dependent calling can diverge is also useful.

  2. The paper provides substantial diagnostic evidence. Budget sweeps, multiple observation replacements, training trajectories, and selected multi-seed and advantage-estimator comparisons make this more informative than a simple benchmark comparison. The appendices also acknowledge important limitations.

Critical weaknesses
  1. The experiments do not sufficiently isolate what RL improves or why tool use fails. The protocol comparison bundles several changes. Although Section 3 describes conditions differing by exactly one variable, Appendix D acknowledges differences in serialization, permitted turns, continuation budgets, and format familiarity. The reported penalty therefore establishes a failure of the tested interface-and-rollout configuration, rather than isolating serialization itself. Fixed-turn, matched-budget, force-answer, and schema-only adaptation controls are needed to distinguish these explanations. The training results also require more careful attribution. In Table 3, RL changes V* full-image accuracy from 73.8% to 73.3%, while tool-protocol accuracy rises from 40.8% to 73.3%. This supports recovery from protocol-induced degradation in this setting, but does not establish a broader account of what tool-use RL learns. A matched answer-only RL baseline would help separate generic post-training effects from learning specifically induced by visual tool interaction. Furthermore, the synthetic training pools have no held-out split, and the reported trajectory accuracies come from training batches. Consequently, increasing no-tool accuracy cannot distinguish improved visual reading from memorization or shortcuts. Appendix D also acknowledges that the attempted tool-necessary construction did not establish necessity. These limitations weaken the proposed explanation of collapse, even though the collapse trajectories themselves remain valid observations.
  2. The budget-gate and causal-credit claims are stronger than the evidence supports. The budget sweeps demonstrate that restricting resolution can increase an oracle crop’s relative value. They do not establish that exceeding the working budget is necessary for useful cropping: crops can also reduce distractors or change attention allocation. The contribution should therefore be positioned as a quantitative characterization of budget sensitivity, rather than a general condition determining whether tools can help. Several details also require reconciliation. Table C.2 reports a content-causal share of approximately 0.94–0.99 for the 7B base model on V*, whereas Section 8 states that no substantially nonzero causal share was observed on below-budget benchmarks. Additionally, the paper juxtaposes its correctness-flip rate with MED’s 0.22–0.30 contribution ratio, although MED defines that ratio through training-time performance drift; these are different estimands and are not directly comparable. Finally, an unchanged greedy answer demonstrates a lack of marginal correctness dependence under the chosen intervention, rather than proving that the evidence was unused. Redundant evidence and confidence changes are not captured. Empirical nesting within probe-style credit demonstrates disagreement between measurements, but does not independently establish that all probe-only credits are re-asking artifacts. Given existing work on tool-effect attribution and observation replacement, the manuscript needs a sharper statement of its incremental diagnostic contribution.
Questions for authors
  1. How much of the protocol penalty remains with matched continuation budgets, fixed turns, forced final answering, and schema-only adaptation?

  2. What do held-out checkpoint evaluations and a matched answer-only RL baseline show about intrinsic improvement, protocol adaptation, and genuine crop-based correction?

  3. How should Table C.2 be reconciled with the below-budget claim in Section 8, and why are the paper’s correctness-flip rate and MED’s differently defined contribution ratio compared numerically

Reviewer_sLahOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
1(−2)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why "crop-and-zoom" tools—trained via reinforcement learning—often fail to effectively improve visual reasoning performance, despite high invocation frequencies and accurate spatial localization. The authors focus on three key factors: the model’s operational pixel budget, the overhead introduced by tool-invocation protocols, and the credit-assignment mechanism based on the content returned by the tools. Through experiments across various vision-language models and benchmarks, the study reveals that a large proportion of tool invocations contribute little to the final predictions. Furthermore, the authors propose an evaluation metric based on counterfactual ablation to determine whether the cropped images returned by tools actually influence the model’s predictions. They also analyze why reinforcement learning struggles to foster the ability to use tools selectively.

Critical strengths
  1. Provides a valuable decomposition of the failure mechanisms in visual tool usage. The paper distinguishes between the potential value of additional visual information and the overhead introduced by tool interfaces. Through controlled experiments involving varying pixel budgets and image-presentation protocols, it uncovers significant performance disparities often overlooked in conventional evaluations of tool usage. These findings offer useful guidance for the design and assessment of visual agents.
  2. Introduces an effective diagnostic method to assess whether tool-returned content is genuinely utilized. The counterfactual ablation analysis goes beyond simple metrics—such as invocation frequency or localization accuracy—to examine whether the returned visual content actually influences the model’s predictions. Notably, the study finds that out of 138 "correct" tool invocations, only 5 result in a change in the model’s prediction when the cropped image is replaced. This discovery is highly insightful and highlights significant limitations in current training objectives for tool usage.
Critical weaknesses
  1. The specific mechanism by which the tool-use protocol leads to performance degradation has not been sufficiently isolated. The observed performance gap between standard dual-image inputs and tool-use inputs is compelling. However, these two settings differ not only in their serialization format but also in the nature of the interaction process—specifically, the inclusion of additional interaction turns and the potential for repeated tool invocations. Consequently, the current experiments cannot definitively determine whether the performance degradation stems primarily from the protocol representation itself or from the interaction behaviors it necessitates. This distinction is crucial for the reliability and practical significance of the paper’s second core finding.
  2. There is a lack of comprehensive end-to-end evidence to support the generalizability of conclusions regarding when cropping is effective. While the "pixel-budget" analysis effectively demonstrates how resolution constraints influence the benefits of cropping, cropping may also yield gains through strategic region selection and the allocation of attention. For instance, even under a nominal 12.8M pixel budget, the inclusion of additional cropped images continues to enhance model performance. More importantly, the primary comparative experiments rely on crops generated using ground-truth annotated regions. It remains unclear whether these findings would hold if the model were to autonomously determine its own cropping strategy. This limits the persuasiveness of the paper’s claims when applied to practical, reinforcement learning (RL)-based visual agents.
Questions for authors
  1. Can the authors isolate the performance loss inherent to the tool-use protocol itself from the effects of repeated tool invocations by controlling for the number of interaction turns and the total generation budget? To what extent does the performance degradation persist under these controlled conditions?
  2. Given that additional cropped images provide performance gains even within a 12.8M pixel budget, how do the authors define the scope of applicability for the "pixel-budget gate"? Furthermore, do the paper’s primary findings regarding the protocol still hold when the cropped regions are generated autonomously by the model, rather than derived from ground-truth annotations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.