- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper highlights the visual insensitivity problem, where VLMs generate answers without relying on visual evidence in reality despite achieving high accuracy on the VQA benchmark. To address this problem, the authors propose VisIO, which enables adaptive visual intervention optimization.
Critical strengths
- The authors conduct extensive experiments on three benchmarks (LongVideoReason, LongVideoBench, and VideoMME).
- The authors provide a well-representative and comprehensive figure to help with understanding.
- Visual Insensitivity Score (VIS) assesses the proportion of correct answers in actual images, even when the input is not perfect (e.g., irrelevant). This highlights the issue that high VQA accuracy does not necessarily imply strong visual grounding.
- VisIO frozens VLM and only learns the visual contribution values for each attention head.
Critical weaknesses
- As I mentioned above, one of its strengths is that it has been evaluated on three video benchmarks. However, I think the authors need to show generalization to the image modality.
- While constructing data without human annotation is efficient, it's essential to have human evaluation to ensure that the generated videos and questions accurately capture the intended meaning and feel natural.
- Since VIS calculations exclude invalid cases (e.g., refusals, irrelevant responses, etc.), it's also a good to report the number or percentage of such cases separately.
- While the question of whether VLMs respond based on visual evidence has been explored in previous studies, VIS offers only a limited new perspective as a metric for quantifying this. So, I think the core contribution of this paper lies in the learning techniques introduced in VisIO, which set it apart from traditional visual intervention methods, and this distinction deserves greater emphasis.
- Please more clearly distinguish VIS from previous grounding metrics and emphasize how VisIO's supervision and adaptive intervention optimization differ from existing visual intervention methods.
Questions for authors
- There are some hidden texts that I don't understand (lines 076, 111, 128, 206, ...). Was this intended? Please explain me about that.
- Since VIS calculations exclude invalid cases (e.g., refusals, irrelevant responses, etc.), it's also a good to report the number or percentage of such cases separately.
6 comments
W5.1. Difference between VIS and grounding metrics
Existing grounding evaluations typically require annotations of the relevant visual evidence [1,2], while VIS can indicate potential grounding failures by measuring visual sensitivity without grounding annotations. VIS therefore provides a simple diagnostic of visual reliance and can be directly applied to various VQA benchmarks, including those without grounding annotations.
We have clarified this distinction in Related Work (Lines 117–121, highlighted in blue).
W5.2. Difference between VisIO and visual intervention methods
Supervision strategy: Existing methods typically use labeled data for determining intervention parameters [3–6], whereas VisIO:
- requires only a small set of unlabeled videos;
- self-constructs supervision through answer-flipping visual counterfactual pairs; and
- automatically evaluates candidate pairs based on visual responsiveness and intervention potential.
Optimization strategy:
- Conventional visual intervention methods use manually specified or fixed intervention parameters [3–6], whereas VisIO automatically learns adaptive intervention parameters.
- VisIO performs intervention-aware pair selection for optimization, selecting counterfactual pairs with high intervention potential rather than directly optimizing over all examples.
- VisIO explicitly improves sensitivity to contrastive visual changes by jointly optimizing counterfactual pairs instead of individual examples.
These designs are specifically introduced for intervention-aware optimization and have not been explored in prior visual intervention methods. Overall, they make VisIO an annotation-free, adaptive, and data-efficient framework for strengthening visual reliance. We have highlighted these distinctions in Related Work (Lines 126–132, highlighted in blue).
[1] Xiao J, Yao A, Li Y, et al. Can I Trust Your Answer? Visually Grounded Video Question Answering. CVPR 2024.
[2] Villa A, Léon J, Soto A, et al. Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models. CVPR 2025.
[3] Sarkar S, et al. Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression. EMNLP 2025.
[4] He J, et al. Cracking the Code of Hallucination in LVLMs with Vision-Aware Head Divergence. ACL 2025.
[5] Su J, et al. Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention. ACL 2025.
[6] Wu J, et al. SHARP: Steering Hallucination in LVLMs via Representation Engineering. EMNLP 2025.
W4. Emphasize VisIO as the primary methodological contribution
Thank you for your constructive suggestion. We have revised the contribution statements (Lines 99–106, highlighted in blue) to emphasize VisIO as our primary methodological contribution. We quote the revised statements below.
1) Our primary methodological contribution is VisIO, a method for explicitly improving VLMs' visual reliance. It automatically learns adaptive visual intervention parameters, in contrast to conventional methods that rely on manually specified intervention strengths.
2) VisIO does not require labeled VQA data for optimization. It constructs its own supervision through answer-flipping visual counterfactual pairs using only a small set of unlabeled target videos, and effectively selects pairs with high intervention potential for optimization.
3) We introduce VIS, a model- and sample-specific metric for measuring visual sensitivity. It can identify potential visual grounding failures across various VQA benchmarks without requiring grounding annotations.
Q1. Hidden texts
We checked all the indicated lines carefully and did not find any hidden text. It would be helpful if the reviewer could provide further clarification, such as the specific text they are referring to.
Follow-up on Our Rebuttal
Dear Reviewer,
Thank you again for your review. As the discussion period is ending soon, we would like to check whether our responses have addressed your concerns. Since our initial response, we have added additional experiments on image-based VQA, human evaluation of the constructed data, analysis of invalid predictions, and further clarification of the distinctions between VIS/VisIO and prior work. We have also updated the manuscript accordingly, with the revised content highlighted in blue for ease of reference.
We would be happy to answer any further questions or conduct additional analyses that you think would be helpful. If our responses have resolved your concerns, we would greatly appreciate it if you would consider updating your score.
Best regards, The authors
We thank the reviewer for the helpful comments.
W1. Generalization to image modality
We report results on three image-based VQA benchmarks in the table below. VisIO consistently improves VQA accuracy while maintaining low VIS, showing that VisIO generalizes effectively beyond video to image-based VQA.
Image-based VQA
Video-based VQA
Comparing the above two tables, we highlight that VIS is notably higher on video than on image benchmarks (average VIS: 11.30 vs. 4.85 for Qwen3). This suggests that visual insensitivity is more substantial in video VQA than in image VQA, motivating our primary focus on the video setting.
We have added this analysis to the revised manuscript (Lines 679–697, highlighted in blue), with additional implementation details provided in Appendix G.
W2. Human evaluation of constructed data
Thank you for your valuable suggestion. We conducted a human evaluation of the constructed video-based VQA samples. Participants assessed three aspects: 1) whether the visual evidence supports the reference answer, 2) whether the intended edit is successfully implemented, and 3) how natural the visual sequence appears.
Each participant was randomly assigned 15 samples for evaluation. We distributed the survey through public AI research community channels and via email. Most participants were students and researchers in computer vision and AI, providing relevant expertise for evaluating the quality of the constructed VQA samples. In total, we collected 286 completed sample evaluations from 21 participants. The results are summarized as follows:
These results provide human evidence for the answer validity, editing fidelity, and overall naturalness of the automatically constructed data. More implementation details are provided in Appendix A.4 (highlighted in blue).
W3&Q2. Report the percentage of invalid cases
Please refer to our response to Reviewer XkQb (W3&Q4) or Appendix F (Lines 1075–1095), where we report the invalid prediction rates under different visual conditions. The rates remain low (≤1.81%) both before and after VisIO, indicating that the reduction in VIS is not driven by invalid outputs.