acceptodds
Under review as a conference paper at ICLR 2027

Vision as Efficient Information Gathering

Abstract

Modern vision-language models perceive every pixel uniformly across the frame(s), spending compute on regions irrelevant to the task. While dynamic-resolution and adaptive-tokenization methods adapt the token budget to the input, they remain task-agnostic and must still process the full image at least once. We introduce ActiveLook, which casts visual perception as a decision-making problem. Starting from a coarse glance, we learn a policy that actively decides, turn by turn, where to look (bounding box, page index, temporal window) and how much to look (scale, frame rate), until it has gathered enough evidence to answer, optimizing accuracy per FLOP. The obstacle is supervision: ground-truth look sequences do not exist. We bootstrap this from a frozen teacher's rollouts, keeping trajectories that verifiably lead to the correct answer, and augment them with synthetic trajectories derived from ground-truth annotations. We distill these trajectories with KL-anchored supervised fine-tuning, followed by reinforcement learning with a compute reward that favors efficient looks across the action space: number of looks, bounding box, scale, temporal window, and frame rate. We gate the reward on correctness and schedule the gate to avoid sparse rewards early in training. ActiveLook learns a single policy spanning images, multi-page documents, and videos, that advances the accuracy-FLOPs Pareto frontier across all regimes: matching the base model's accuracy with less compute on images and long documents, and less on video. At matched input budgets, ActiveLook is points more accurate on average while using 27% less compute.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, active perception, adaptive perception, efficient visual reasoning, reinforcement learning, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YGFUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce ACTIVELOOK, which casts visual perception as a decision-making problem. Rather than making a dedicated glance at first, it starts from the coarse glance, and learns a policy that actively decides where to look and how much to look. To achieve this, this paper suggests to conduct distillation based SFT with reinforcement learning, with some techniques like trajectory augmentation and using annotated trajectories for SFT, and reward scheduling for RL.

Critical strengths

S1. Well-motivated and compelling approach to visual perception. The paper presents a compelling framework for training VLMs to selectively acquire visual information, learning both where to look and how much visual detail to acquire.

S2. Strong empirical performance across diverse visual tasks. ACTIVELOOK achieves strong results across a broad range of image, document, and video benchmarks, demonstrating the effectiveness and generality of the proposed approach.

Critical weaknesses

W1. Insufficient explanation of important implementation details

The paper omits several low-level details that are important for understanding and reproducing the proposed method. Providing additional clarification on the following points would substantially improve the technical completeness of the paper.

  • Action omission by the base model. It is unclear how the base model decides to omit an action. If this mechanism relies on the model's native tool-calling functionality, the authors should describe the implementation in more detail, including how the action arguments are defined.
  • Distribution of the coarse-glance parameter ss. The paper does not explain how the distribution of the coarse-glance parameter ss is selected during training, thought it might be important to understand the evaluation results. The authors should clarify the sampling strategy and provide the rationale behind this choice.
  • Selection of ρhi\rho_{\mathrm{hi}} and ρlo\rho_{\mathrm{lo}}. The paper should explain how these hyperparameters are chosen and why the reported values are appropriate. In addition, I believe a sensitivity analysis with respect to ρhi\rho_{\mathrm{hi}} and ρlo\rho_{\mathrm{lo}} would be valuable for demonstrating the robustness of the method.

W2. Insufficient justification for several design choices

Several components of the proposed training and scheduling pipeline are introduced without sufficient empirical justification.

  • Effect of additional SFT techniques. The paper does not clearly isolate the contribution of the additional techniques used during SFT, such as annotation and data augmentation. Ablation studies would help quantify the individual contribution of these components.
  • Necessity of the scheduling technique. I am not fully convinced that the proposed scheduling technique is necessary based on the current evidence. Although Figure 6 is intended to demonstrate its effectiveness, the curves appear to overlap substantially, and α=0\alpha=0 even seems to provide a better trade-off in the low-budget regime. At minimum, the authors should provide separate plots for each benchmark so that the effect of scheduling can be examined more clearly. A more explicit quantitative comparison would also help establish whether the additional scheduling mechanism provides a consistent benefit.

W3. Additional baselines are needed

The paper discusses conventional efficiency approaches such as token pruning and token merging, and argues that ACTIVELOOK is more effective because it can operate along multiple axes. However, this claim is not sufficiently supported by the current experiments.

I recommend including representative token-pruning and/or token-merging methods for each benchmarks as additional baselines under comparable computational budgets. Such comparisons would more directly demonstrate the advantages of ACTIVELOOK over existing efficiency-oriented approaches and substantiate the paper's claim regarding its multi-axis adaptation capability.

Questions for authors

Q1. What is the role of setting c=3c=3 in the experimental setup?

Q2. Could the authors report the Figure 6 ablation results separately for each benchmark? This would make it easier to understand how the scheduling technique affects performance across different benchmarks and whether its impact is consistent.

Q3. Could the authors further clarify the fundamental differences between ACTIVELOOK and existing tool-calling approaches that incorporate a cost penalty? Conceptually, ACTIVELOOK appears related to training a model to call a specific vision tool while penalizing tool usage, with a bag of tricks. It would therefore be helpful for the authors to articulate more clearly what is fundamentally novel about ACTIVELOOK beyond this formulation, and how it differs from prior cost-aware tool-calling methods.

Reviewer_GBNHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ActiveLook, an adaptive computation method for VLMs that aims to learn where and when to look for more details in an initially coarse image. The model is trained in multiple stages, first using SFT (as an anchor) and then GRPO, with a multi-reward objective. The evaluations are conducted on a small set of benchmarks.

Critical strengths
  • The general direction of adaptive computation makes sense and is interesting
Critical weaknesses
  • The paper is of limited novelty. There is a vast body of works that already performs zoom+crop with and without RL techniques. As far as I can tell there is no notable difference from these prior works, and if any, currently the paper doesn't contrast them properly. Just to offer a few examples: [1] Learning When and Where to Zoom with Deep Reinforcement Learning, CVPR 2020 (not VLM specific, but same decisional goals), [2] AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition, CVPR 2026, [3] Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models, 2026 (pretty much the same accuracy-gated computer reward), [4] Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs, 2025, [5] Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning, 2025, [6] AdaTooler-V: Adaptive Tool-Use for Images and Videos, 2025, [7] FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learning, 2025, etc. This is by no means an exhaustive list so I encourage the authors to look beyond it.

  • There is very limited comparison with state-of-the-art. Based solely on this (and the bellow issues) it's not really possible to asses how well the proposed approach works.

  • The base model protocol doesn't match what Qwen3-VL-4B officially reports.

  • The external baselines numbers are confusing and appear degraded (different backbnes, different numbers from the ones published, unclear how the settings differ).

  • FLOPs' accounting is largely unspecified, not sreally sure how they were computed for all the reference methods based on the text.

Questions for authors

In addition to the weaknesses section (please see above):

  • What does the plain single-pass Qwen3-VL-4B at native resolution score on each benchmark? What about with Qwen's official zoom-tool prompt?
  • What does SFT on the same data and teacher answers without look actions get?
  • Why are the baseline numbers below their published numbers? All comparisons should be done in matched setting (ideally in the ones from the original reference works).
Reviewer_tkhJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ACTIVELOOK, a unified policy for efficient visual information gathering across images, multi-page documents, and videos. Starting from a coarse glance, the model adaptively selects where to look, how much detail to acquire, and when to answer. Training combines supervised fine-tuning on teacher-generated and annotation-derived trajectories with reinforcement learning using an accuracy-gated compute reward. Experiments across six benchmarks demonstrate improved accuracy–FLOPs trade-offs over the base model and existing efficient-perception methods.

Critical strengths
  1. A unified policy across diverse visual tasks. The same policy handles images, multi-page documents, and videos through adaptive spatial and temporal observation. Improvements across six benchmarks support the applicability of the approach across different visual settings, strengthening the empirical contribution.
  2. Reward design that balances efficiency and useful exploration. ACTIVELOOK combines a correctness-gated compute reward with a training schedule that preserves early exploration and gradually rewards efficient, correct trajectories. The ablations show that removing correctness gating causes the policy to stop acquiring additional evidence, supporting the importance of this design for reducing computation while retaining useful visual observations.
Critical weaknesses
  1. Incomplete baseline controls for the efficiency claims. The reference base model uses a multi-turn tool loop, while a standard single-pass evaluation of the base model is not reported. Comparisons with prior methods also involve differences in backbone and input protocol. Although the SFT/RL ablations are informative, these gaps limit attribution of the accuracy–FLOPs advantage to the proposed method and leave its benefit over conventional inference insufficiently established.
  2. Limited validation of the observation policy’s robustness and contribution. The compute-reward thresholds lack sensitivity analysis, and the video experiments do not isolate the benefit of joint spatial–temporal control over temporal-only zooming. Furthermore, the visual-search evaluation does not cover challenging benchmarks such as HRBench-8K or VisualProbe-Hard. These gaps leave uncertainty about which design choices drive the gains and whether the accuracy–efficiency trade-off remains favorable under more demanding perception and exploration requirements.
Questions for authors
  1. Choice and robustness of compute-reward thresholds. What motivates setting ρ_lo = 3.5 and ρ_hi = 12? Equation 3 gives identical compute rewards to trajectories costing between one and 3.5 times the single-pass reference. Can the authors explain this choice and report how different thresholds affect the accuracy–FLOPs frontier?
  2. Contribution of joint spatial–temporal control. Section 2 emphasizes joint spatial–temporal adaptation, but the video experiments do not isolate its benefit over temporal-only zooming. Can the authors compare the full model with a temporal-only video variant trained under matched conditions, reporting their accuracy–FLOPs frontiers to quantify the additional benefit of spatial control?
  3. Backbone-controlled comparisons. ACTIVELOOK uses Qwen3-VL-4B, whereas the original VisionThink and VideoZoomer models use Qwen2.5-VL-7B. Can the authors provide comparisons using a common backbone with matched training data, budgets, and evaluation protocols, to isolate the training method’s contribution from differences in base-model capabilities and size?
  4. Single-pass inference baseline. Section 5.1 evaluates the zero-shot base within a multi-turn tool loop, and Section 5.3 reports an 83–88% look rate on V*Bench across scales. Can the authors also report standard single-pass accuracy and FLOPs across the same input scales, including full resolution? This would establish the efficiency advantage over conventional inference and quantify savings from avoiding unnecessary tool calls.
  5. Efficiency on more demanding visual tasks. Section 5.3 reports that only 3% of V*Bench examples trigger additional observation at full resolution. Can the authors evaluate on HRBench-8K and VisualProbe-Hard, reporting accuracy, total FLOPs, and interaction counts? This would test whether the accuracy–efficiency gains persist when finer perception or more extensive exploration is required.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.