acceptodds
Under review as a conference paper at ICLR 2027

Measuring One-Step Visual Acquisition: Probe Shifts and Outcome-Supervised Value

Abstract

Visual-acquisition scores depend on how model responses are elicited and how task outcomes are scored. We examine these measurement dependencies using same-state, one-step crop interventions. Empirical Visual Intervention Effect (VIE) estimates an outcome-supervised continuation-reward contrast and serves as a costly offline reference. The comparator, termed realized information gain (RIG), computes KL divergence between four-choice readout distributions before and after acquisition under a fixed probe interface. On 120 constructed questions and 360 saved Qwen3-VL-32B-Instruct states, frozen empirical-VIE and RIG selections achieve 96.25% and 89.31% fresh stored-key agreement, respectively: a +6.94 percentage-point difference (95% question-cluster CI [4.10, 10.28]), conditional on realized discovery selections. A replication with Mistral-Small-3.2-24B-Instruct-2506, using its own states and separately sampled continuations on the same questions and stored keys, yields +10.56 points [6.39, 14.86]. Task construction limits these comparisons: 13 of 30 Geometry questions have no exact correct option, and source-conditioned option rules recover all 120 stored keys without images, exposing a shortcut without establishing model use of it. Exact-value rescoring with the original inputs and selections fixed yields a Qwen difference of +4.65 points [2.43, 7.22]; this is not evaluation on repaired tasks. A pre-specified 12-question development study also finds selection sensitivity to the tested prompt and readout changes, but fresh outcomes for variant-selected actions remain unmeasured. Learned-policy comparisons do not establish VIE-based superiority. The contribution is a controlled measurement protocol and a diagnosis of reward, readout and selection dependencies, rather than a deployable acquisition policy or a general ranking of acquisition principles.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multimodal reasoning, visual acquisition, model evaluation, belief probing, value of information

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QbNCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper compares Realized Information Gain (RIG) and Visual Intervention Effect (VIE) for selecting image regions during multimodal reasoning. The authors report that VIE-based selection achieves higher downstream accuracy than RIG across several experiments. The paper also includes extensive diagnostic analyses and attempts to learn a value predictor, although the latter does not demonstrate a clear advantage.

Critical strengths

The separation between discovery and fresh evaluation is reasonable. The authors also provide extensive diagnostic results and acknowledge several limitations. However, these do not resolve my main concerns about the validity and significance of the study.

Critical weaknesses
  1. Limited novelty and questionable research contribution. The paper reads more like a collection of heavily AI-assisted experimental reports than a focused scientific investigation. The central observation is not particularly novel, and the excessive post-hoc analyses add little to the main contribution.
  2. Serious issues with experimental validity. The authors acknowledge that all 120 stored answer keys can be recovered through rules that do not require visual information, and 13 Geometry questions contain invalid answer options. Furthermore, most states have tied VIE-optimal regions, making the interpretation of meaningful visual region selection questionable. Although additional diagnostics partially address these issues, the fundamental concerns remain. Overall, I find the scientific contribution weak and the experimental evidence insufficient for acceptance.
Questions for authors
  1. Given that all 120 stored answer keys can be recovered without visual information, how can the authors establish that the reported improvements reflect meaningful visual acquisition rather than dataset shortcuts?

  2. With only 8 out of 360 states having a unique VIE-optimal region, what evidence supports the reliability and significance of the proposed region-selection criterion?

  3. Beyond the well-established distinction between information gain and task utility, what is the actual novel contribution of this work, particularly given that the learned critic fails to demonstrate a clear advantage?

Reviewer_SnurOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents an experimental approach to compare two one-step region-value measurements for scoring visual region selection, including realized information gain (RIG) and visual intervention effect (VIE). RIG is to measure how the answer-probe distributions change before and after inserting a crop, while VIE measures how much inserting a crop helps get the answer correct. The experimental study covers multiple VLMs (QWen, Mistral) and uses 120 MCQs with four categories. The result shows that RIG and VIE are not interchangeale criteria.

Critical strengths
  1. The paper addresses an important question in VQA: how VLMs should decide which image region to inspect to improve accuracy.

  2. This paper takes a controlled one-step forced-LOOK setting with saved-state branching and fresh continuations. The study shows an interesting finding: visual crops that strongly alter a model's predictions doesn't necessarily improve final accuracy.

  3. The proposed experimental approach can be generalize to other tasks, such as probing active visual reasoning and decision making behaviors in other AI models.

Critical weaknesses
  1. The main technical contribution of the work lies in the experimental approach and the results. This makes the novelty limited and the contribution incremental.

  2. Although the paper acknowledged the limitations of the dataset (non-visual shortcut and invalid questions) and took some follow up steps, the issue is not fully addressed

  3. The calculation of empirical VIE is computationally expensive, and the proposed pre-aquisition neural critic fails to outperform RIG predictors.

  4. The conclusion drawn from the experimental study, i.e., using VIE (accuracy driven measurement) strategy can outperform RIG (prediction change driven measurement) is intuitive. Although the paper includes subgroup analysis, its conclusion would be stronger if the analysis can be translated into the insights into when VIE outperforms, similar to, or underperforms RIG? This could help understand when the change of prediction is a useful proxy of accuracy and when it is not.

Questions for authors

Please see the list of weaknesses.

Reviewer_hDi6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes VIE as an offline reference metric for outcome supervision. Through the experimental paradigm of freezing the reasoning state and separating discovery from fresh evaluation, the experimental results show that RIG (posterior KL information gain), which measures the change in model answer beliefs, is not equivalent to the actual downstream task benefits. The local regions of the images selected by the two will bring significantly different reasoning accuracy rates.

Critical strengths
  1. The proposed work employs a counterfactual experimental paradigm with complete separation between discovery-stage candidate screening and fresh independent evaluation, which mitigates inflated performance estimates due to sample reuse and delivers a rigorous offline benchmarking framework for cross-comparison of multimodal visual acquisition strategies.
  2. It empirically demonstrates that belief-shift-based information gain metrics differ significantly from actual downstream task gains, revealing the inherent limitation of the dominant reliance on posterior distribution shift metrics in the field of active vision.
Critical weaknesses
  1. A portion of the geometry problems have no exact correct option, and there exist text-only shortcuts that allow answer recovery without visual input, which limits the semantic validity of the conclusions and their cross-task generalizability.
  2. The core metric VIE depends on extensive counterfactual generation rollouts and incurs prohibitively high computational costs.
  3. The experiments are confined to a one-step forced-LOOK visual acquisition paradigm, do not cover multi-step sequential active visual exploration scenarios, and are only validated on a four-alternative multiple-choice format, resulting in a gap between the task setup and real-world reasoning scenarios.
Questions for authors

See above Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.