- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper compares Realized Information Gain (RIG) and Visual Intervention Effect (VIE) for selecting image regions during multimodal reasoning. The authors report that VIE-based selection achieves higher downstream accuracy than RIG across several experiments. The paper also includes extensive diagnostic analyses and attempts to learn a value predictor, although the latter does not demonstrate a clear advantage.
Critical strengths
The separation between discovery and fresh evaluation is reasonable. The authors also provide extensive diagnostic results and acknowledge several limitations. However, these do not resolve my main concerns about the validity and significance of the study.
Critical weaknesses
- Limited novelty and questionable research contribution. The paper reads more like a collection of heavily AI-assisted experimental reports than a focused scientific investigation. The central observation is not particularly novel, and the excessive post-hoc analyses add little to the main contribution.
- Serious issues with experimental validity. The authors acknowledge that all 120 stored answer keys can be recovered through rules that do not require visual information, and 13 Geometry questions contain invalid answer options. Furthermore, most states have tied VIE-optimal regions, making the interpretation of meaningful visual region selection questionable. Although additional diagnostics partially address these issues, the fundamental concerns remain. Overall, I find the scientific contribution weak and the experimental evidence insufficient for acceptance.
Questions for authors
-
Given that all 120 stored answer keys can be recovered without visual information, how can the authors establish that the reported improvements reflect meaningful visual acquisition rather than dataset shortcuts?
-
With only 8 out of 360 states having a unique VIE-optimal region, what evidence supports the reliability and significance of the proposed region-selection criterion?
-
Beyond the well-established distinction between information gain and task utility, what is the actual novel contribution of this work, particularly given that the learned critic fails to demonstrate a clear advantage?