acceptodds
Under review as a conference paper at ICLR 2027

Correct Yet Insensitive to Vision: Adaptive Intervention to Improve Visual Reliance in VLMs

Abstract

While vision-language models (VLMs) continue to improve on visual question answering (VQA) benchmarks, we identify a critical limitation: an apparently correct answer may not derive from visual evidence. We introduce the Visual Insensitivity Score (VIS) to quantify this behavior and surprisingly find that many correct answers remain unchanged when visual input is replaced or removed. This reveals substantial visual insensitivity in VLMs that conventional VQA accuracy fails to capture. To improve visual reliance, existing methods typically employ visual interventions with manually specified strengths, without explicitly accounting for whether model answers are grounded in visual evidence. To address this limitation, we propose Adaptive Visual Intervention Optimization (VisIO), which explicitly optimizes lightweight interventions to enforce reliance on answer-determining visual evidence rather than other contextual cues. VisIO achieves this by constructing language-preserving, answer-flipping visual counterfactual pairs, where the linguistic context and most visual content remain unchanged while controlled visual changes flip the correct answer. Importantly, VisIO constructs such pairs using a small set of unlabeled target videos and optimizes only hundreds of head-wise parameters before inference. Across multiple VLMs and video VQA benchmarks, VisIO consistently improves accuracy while reducing visual insensitivity and strengthening reliance on visual evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language model, visual insensitivity, visual quesiton answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5vmoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper highlights the visual insensitivity problem, where VLMs generate answers without relying on visual evidence in reality despite achieving high accuracy on the VQA benchmark. To address this problem, the authors propose VisIO, which enables adaptive visual intervention optimization.

Critical strengths
  1. The authors conduct extensive experiments on three benchmarks (LongVideoReason, LongVideoBench, and VideoMME).
  2. The authors provide a well-representative and comprehensive figure to help with understanding.
  3. Visual Insensitivity Score (VIS) assesses the proportion of correct answers in actual images, even when the input is not perfect (e.g., irrelevant). This highlights the issue that high VQA accuracy does not necessarily imply strong visual grounding.
  4. VisIO frozens VLM and only learns the visual contribution values for each attention head.
Critical weaknesses
  1. As I mentioned above, one of its strengths is that it has been evaluated on three video benchmarks. However, I think the authors need to show generalization to the image modality.
  2. While constructing data without human annotation is efficient, it's essential to have human evaluation to ensure that the generated videos and questions accurately capture the intended meaning and feel natural.
  3. Since VIS calculations exclude invalid cases (e.g., refusals, irrelevant responses, etc.), it's also a good to report the number or percentage of such cases separately.
  4. While the question of whether VLMs respond based on visual evidence has been explored in previous studies, VIS offers only a limited new perspective as a metric for quantifying this. So, I think the core contribution of this paper lies in the learning techniques introduced in VisIO, which set it apart from traditional visual intervention methods, and this distinction deserves greater emphasis.
  5. Please more clearly distinguish VIS from previous grounding metrics and emphasize how VisIO's supervision and adaptive intervention optimization differ from existing visual intervention methods.
Questions for authors
  1. There are some hidden texts that I don't understand (lines 076, 111, 128, 206, ...). Was this intended? Please explain me about that.
  2. Since VIS calculations exclude invalid cases (e.g., refusals, irrelevant responses, etc.), it's also a good to report the number or percentage of such cases separately.
6 comments

We thank the reviewer for the helpful comments.

W1. Generalization to image modality

We report results on three image-based VQA benchmarks in the table below. VisIO consistently improves VQA accuracy while maintaining low VIS, showing that VisIO generalizes effectively beyond video to image-based VQA.

Image-based VQA

MMBenchV*MMStarAvg.
ACC ↑   VIS ↓ACC ↑   VIS ↓ACC ↑   VIS ↓VIS ↓
Qwen389.1     9.6383.7     1.6666.8     3.274.85
+VisIO90.7     8.2285.9     1.7868.1     2.894.30

Video-based VQA

LongVideoReasonLongVideoBenchVideoMMEAvg.
ACC ↑   VIS ↓ACC ↑   VIS ↓ACC ↑   VIS ↓VIS ↓
Qwen369.8     18.949.5     6.558.2     8.511.30
+VisIO72.1     14.353.1     4.560.6     4.87.87

Comparing the above two tables, we highlight that VIS is notably higher on video than on image benchmarks (average VIS: 11.30 vs. 4.85 for Qwen3). This suggests that visual insensitivity is more substantial in video VQA than in image VQA, motivating our primary focus on the video setting.

We have added this analysis to the revised manuscript (Lines 679–697, highlighted in blue), with additional implementation details provided in Appendix G.

W2. Human evaluation of constructed data

Thank you for your valuable suggestion. We conducted a human evaluation of the constructed video-based VQA samples. Participants assessed three aspects: 1) whether the visual evidence supports the reference answer, 2) whether the intended edit is successfully implemented, and 3) how natural the visual sequence appears.

Each participant was randomly assigned 15 samples for evaluation. We distributed the survey through public AI research community channels and via email. Most participants were students and researchers in computer vision and AI, providing relevant expertise for evaluating the quality of the constructed VQA samples. In total, we collected 286 completed sample evaluations from 21 participants. The results are summarized as follows:

  1. The visual evidence was judged to support the reference answer in 92.3% of the evaluations.
  2. The intended edit was judged to be successfully implemented in 88.5% of the evaluations.
  3. The mean naturalness rating was 4.13/5.

These results provide human evidence for the answer validity, editing fidelity, and overall naturalness of the automatically constructed data. More implementation details are provided in Appendix A.4 (highlighted in blue).

W3&Q2. Report the percentage of invalid cases

Please refer to our response to Reviewer XkQb (W3&Q4) or Appendix F (Lines 1075–1095), where we report the invalid prediction rates under different visual conditions. The rates remain low (≤1.81%) both before and after VisIO, indicating that the reduction in VIS is not driven by invalid outputs.

W5.1. Difference between VIS and grounding metrics

Existing grounding evaluations typically require annotations of the relevant visual evidence [1,2], while VIS can indicate potential grounding failures by measuring visual sensitivity without grounding annotations. VIS therefore provides a simple diagnostic of visual reliance and can be directly applied to various VQA benchmarks, including those without grounding annotations.

We have clarified this distinction in Related Work (Lines 117–121, highlighted in blue).

W5.2. Difference between VisIO and visual intervention methods

Supervision strategy: Existing methods typically use labeled data for determining intervention parameters [3–6], whereas VisIO:

  1. requires only a small set of unlabeled videos;
  2. self-constructs supervision through answer-flipping visual counterfactual pairs; and
  3. automatically evaluates candidate pairs based on visual responsiveness and intervention potential.

Optimization strategy:

  1. Conventional visual intervention methods use manually specified or fixed intervention parameters [3–6], whereas VisIO automatically learns adaptive intervention parameters.
  2. VisIO performs intervention-aware pair selection for optimization, selecting counterfactual pairs with high intervention potential rather than directly optimizing over all examples.
  3. VisIO explicitly improves sensitivity to contrastive visual changes by jointly optimizing counterfactual pairs instead of individual examples.

These designs are specifically introduced for intervention-aware optimization and have not been explored in prior visual intervention methods. Overall, they make VisIO an annotation-free, adaptive, and data-efficient framework for strengthening visual reliance. We have highlighted these distinctions in Related Work (Lines 126–132, highlighted in blue).

[1] Xiao J, Yao A, Li Y, et al. Can I Trust Your Answer? Visually Grounded Video Question Answering. CVPR 2024.

[2] Villa A, Léon J, Soto A, et al. Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models. CVPR 2025.

[3] Sarkar S, et al. Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression. EMNLP 2025.

[4] He J, et al. Cracking the Code of Hallucination in LVLMs with Vision-Aware Head Divergence. ACL 2025.

[5] Su J, et al. Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention. ACL 2025.

[6] Wu J, et al. SHARP: Steering Hallucination in LVLMs via Representation Engineering. EMNLP 2025.

W4. Emphasize VisIO as the primary methodological contribution

Thank you for your constructive suggestion. We have revised the contribution statements (Lines 99–106, highlighted in blue) to emphasize VisIO as our primary methodological contribution. We quote the revised statements below.

1) Our primary methodological contribution is VisIO, a method for explicitly improving VLMs' visual reliance. It automatically learns adaptive visual intervention parameters, in contrast to conventional methods that rely on manually specified intervention strengths.

2) VisIO does not require labeled VQA data for optimization. It constructs its own supervision through answer-flipping visual counterfactual pairs using only a small set of unlabeled target videos, and effectively selects pairs with high intervention potential for optimization.

3) We introduce VIS, a model- and sample-specific metric for measuring visual sensitivity. It can identify potential visual grounding failures across various VQA benchmarks without requiring grounding annotations.

Q1. Hidden texts

We checked all the indicated lines carefully and did not find any hidden text. It would be helpful if the reviewer could provide further clarification, such as the specific text they are referring to.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your review. As the discussion period is ending soon, we would like to check whether our responses have addressed your concerns. Since our initial response, we have added additional experiments on image-based VQA, human evaluation of the constructed data, analysis of invalid predictions, and further clarification of the distinctions between VIS/VisIO and prior work. We have also updated the manuscript accordingly, with the revised content highlighted in blue for ease of reference.

We would be happy to answer any further questions or conduct additional analyses that you think would be helpful. If our responses have resolved your concerns, we would greatly appreciate it if you would consider updating your score.

Best regards, The authors

Reviewer_XkQbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The work addresses an important problem of faithfulness to visual evidence in answers generated by VLMs. It introduces Visual Insensitivity Score which measures the ratio of answers that remain unchanged under perturbation to the visual content such as blank videos, random videos or omission of the visual information i.e text only. In quantifying this, they found that a high VQA accuracy does not correlate with reliance on visual evidence.

To address this problem, it proposes Adaptive Visual Intervention Optimization, a framework that optimizes a lightweight adapter to intervene and learn head-wise scaling parameters to modulate attention and strengthen visual contributions while keeping the VLM frozen. It does so by constructing language-preserving, answer-flipping visual counterfactual pairs from unlabeled videos, selecting informative pairs based on visual responsiveness and intervention potential then using this constructed pairs to optimize the adapter. Experiments were conduction on three video VQA benchmarks(LongVideoBench, Video-MME, LongVideoReason) and four VLMs (VideoZoomer, Qwen3Vl-8B, OneThinker, LLaVA-OneVision-1.5-8B), results show consistent improvements in overall accuracy and reductions in VIS.

Critical strengths
  • The paper formalizes an important distinction between answer correctness and reliance on visual evidence, which is insufficiently captured by conventional VQA accuracy measures and introduces a metric to quantify this gap.

  • The proposed method is lightweight and learns only a small number of scaling parameters without modifying the original VLM weights, allowing for easy adaptation across various methods and models of varying scale. This is further supported via experimental results showing consistent gains across models

  • The idea of counterfactual self-supervision by constructing language-preserving, answer-flipping visual counterfactuals is a compelling part of the paper. Keeping the text-query constant while changing the correct answer creates a supervision signal that cannot be fulfilled by relying on language cues and ignoring visual content. Furthermore, the automatic generation of labels bypasses the need for expensive manual annotation.

Critical weaknesses
  • The method is more of a target-dataset adaptation rather than a fully training-free or benchmark-independent intervention. Furthermore, I have some queries on the sampled training videos and if they overlap with the evaluation set. Line 360 states that 128 unlabeled target videos are sampled for each benchmark, but it is unclear whether these videos are disjoint from the evaluation videos. If the same underlying videos are encountered during adaptation and evaluation, even without using the original VQA labels, this limits the generalization of the method. More importantly, questions remains on whether the method provides a general visual-reliance mechanism or dataset-specific intervention i.e if intervention parameters learned on one dataset generalize to another dataset or to out-of-distribution videos.

  • Visual Insensitivity is a function of both the model and the question. Post-training alters the model's behaviour by scaling the visual contribution of heads. However, the current evaluation largely reports aggregate changes in accuracy and VIS, without directly analyzing how individual samples transition between visually insensitive and visually sensitive behaviour. Since the set of correctly answered samples can change after intervention, a reduction in VIS does not necessarily establish that previously correct but visually insensitive answers have become visually sensitive while remaining correct. A before-and-after transition analysis would provide more direct evidence of the intervention's effect. For example, it would be useful to report the proportions of samples transitioning from insensitive to sensitive and vice versa, together with changes in answer correctness. In particular, distinguishing (correct, insensitive) → (correct, sensitive) from (correct, insensitive) → (incorrect, sensitive) would clarify whether improved sensitivity is accompanied by improved visual grounding.

  • In Appendix E, lines 904–909, it states that a sample is considered visually insensitive only if all four conditions produce valid, identical answers and that "refusals, irrelevant responses, and outputs from which no valid final option can be reliably extracted are treated as invalid predictions and cannot be counted as visually insensitive." This creates a potential issue where a reduction in VIS could partly arise from increased invalid-output rates under perturbed inputs, rather than increased reliance on visual evidence. Invalid prediction rates for each visual condition, before and after VisIO, and a robustness analysis that excludes cases with invalid perturbation responses or reports them separately can mitigate this.

  • This is not a particularly major issue that takes away from the contribution of the work, but model visual insensitivity and use of perturbations such as removal [1] or altering [2] in the visual input space to illustrate this is not new and is a phenomenon that has been established even in early VQA work.
    Authors claim in line 92 -94 "Our main contributions are summarized as follows: 1) We reveal that existing VLMs can suffer from visual insensitivity: an apparently correct answer can remain unchanged even when visual input is altered or removed...". They may wish to soften their claim on this , or clarify how their analysis and findings differ from prior work, particularly in terms of what new insights it provides into the phenomenon of visual insensitivity.

[1] Xiao, Junbin, et al. "Can i trust your answer? visually grounded video question answering." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.
[2] Villa, A., Léon, J., Soto, A., & Ghanem, B. (2025). Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 492-502).

Questions for authors
  • Are the unlabeled target videos used in training overlapping with evaluation videos?
  • What are the cross-dataset transfer or out of domain results using the same learned intervention parameters?
  • Could the authors provide an analysis on the change ratio of sensitive vs insensitive post training?
  • Could the authors report the invalid prediction rates?
3 comments
AuthorsOpenReview ↗

Response to Reviewer XkQb

We appreciate the reviewer's insightful comments.

W1&Q1. The overlap between unlabeled videos used in training and evaluation.

The unlabeled target videos are included in the evaluation, consistent with test-time adaptation and transductive learning settings [1,2], where unlabeled target samples can participate in adaptation.

To assess whether the gains are driven by this overlap, we separately evaluate videos used and not used for VisIO optimization below. VisIO consistently improves ACC and reduces VIS on both subsets, demonstrating that it generalizes effectively to videos unseen during optimization.

Importantly, VisIO uses only a small fraction of the target videos for optimization: 12.8%, 9.57%, and 4.74%, respectively, rather than adapting on the entire target set.

This analysis has been added to Appendix A.2 (highlighted in blue).

Evaluation subsetMethodLongVideoReasonLongVideoBenchVideoMME
Ratio  ACC ↑  VIS ↓Ratio  ACC ↑  VIS ↓Ratio  ACC ↑  VIS ↓
Used for optimizationQwen312.8%  70.2   19.39.57%   49.2   6.74.74%   58.5   8.9
+VisIO12.8%  71.6   14.09.57%   53.3   4.34.74%   60.3   4.1
Not used for optimizationQwen387.2%  69.7   18.790.43%  49.5  6.495.26%  58.2   8.4
+VisIO87.2%  72.2   14.590.43%  53.1  4.695.26%  60.6   4.8

[1] Wang D, et al. Tent: Fully test-time adaptation by entropy minimization. ICLR, 2021.

[2] Zanella M, et al. Boosting vision-language models with transduction. NeurIPS, 2024.

W1&Q2. Cross-dataset transfer

Thank you for this constructive comment. We evaluate cross-dataset transfer in the table below. VisIO consistently improves ACC and reduces VIS. Therefore, the learned intervention parameters are not tied to a specific dataset and generalize effectively across video benchmarks. VisIO does not optimize for dataset-specific answer distributions, but encourages the model to rely more on visual evidence for reasoning.

Our current setting focuses on self-adaptation without external video data. A promising extension is to learn a shared intervention from an external video set and directly apply it across benchmarks, enabling benchmark-independent intervention.

This analysis has been added to Appendix A.1 (highlighted in blue).

MethodOptimization datasetLongVideoReasonLongVideoBenchVideoMME
ACC ↑     VIS ↓ACC ↑     VIS ↓ACC ↑     VIS ↓
Qwen3—69.8     18.949.5     6.558.2     8.5
+VisIOLongVideoReason72.1     14.353.3     5.260.0     5.6
+VisIOLongVideoBench71.7     15.153.1     4.560.4     5.2
+VisIOVideoMME71.2     14.652.4     5.560.6     4.8
VideoZoomer—77.1     67.753.2     35.559.2     25.9
+VisIOLongVideoReason78.8     52.755.3     22.860.4     16.4
+VisIOLongVideoBench79.1     55.354.5     23.160.7     16.0
+VisIOVideoMME78.4     56.253.4     25.661.4     14.4

W2&Q3. Sample-level transition analysis

Before VisIO → After VisIOCorrect, InsensitiveCorrect, SensitiveIncorrect, InsensitiveIncorrect, Sensitive
Correct, Insensitive52.27%44.70%0.00%3.03%
Correct, Sensitive6.18%88.16%0.00%5.65%
Incorrect, Insensitive0.00%6.90%44.83%48.28%
Incorrect, Sensitive1.11%20.30%1.48%77.12%

We conduct a sample-level transition analysis for Qwen3 on LongVideoReason. Among initially (correct, insensitive) samples, 44.70% become (correct, sensitive), while only 3.03% become (incorrect, sensitive). This indicates that VisIO improves visual sensitivity while largely preserving answer correctness, thereby strengthening visual reliance.

This analysis has been added to Appendix A.5 (highlighted in blue).

W3&Q4. Report the invalid prediction rates

We report the invalid prediction rates in the table below. The rates remain consistently low (≤1.81%) and comparable before and after VisIO. These rare invalid predictions have only a small effect on VIS: after excluding samples with invalid perturbation responses, VisIO still consistently reduces VIS across all benchmarks, confirming that the VIS reduction is not driven by invalid outputs.

Invalid outputs mainly reflect generation failures, such as incomplete or malformed responses, rather than meaningful answer changes induced by visual perturbations. Therefore, they should not be interpreted as evidence of visual sensitivity and are excluded from visually insensitive cases.

We have added this analysis to the revised manuscript and provided representative examples in Appendix F (highlighted in blue).

DatasetMethodReal VideoRandom VideoBlank VideoText OnlyOriginal VISVIS (Valid Responses Only)
LongVideoReasonQwen301.26%0.28%0.56%18.919.2
+VisIO01.13%0.30%0.42%14.314.7
LongVideoBenchQwen300.72%0.29%0.86%6.57.2
+VisIO00.63%0.25%0.90%4.55.1
VideoMMEQwen301.81%0.66%1.21%8.58.8
+VisIO01.43%0.63%1.25%4.85.2

The real-video invalid rate is 0, since VIS is evaluated only on samples that are correctly answered with a valid prediction under the real video.

W4. Reframing the contributions

Thank you for your valuable suggestion. We agree that visual insensitivity itself is not a newly discovered phenomenon. We have therefore revised the abstract and introduction (highlighted in blue in the updated manuscript) to avoid such statements.

Specifically, we clarify:

  1. Difference between VIS and relevant metrics. Prior evaluations [3,4] rely on additional grounding annotations of relevant visual evidence, while VIS measures visual sensitivity to indicate potential grounding failures without grounding annotations. VIS therefore provides a simple diagnostic of visual reliance and can be directly applied across various VQA benchmarks, including those without grounding annotations. We have clarified this distinction in Related Work (Lines 117–121, highlighted in blue).

  2. Reframing the contributions. We have revised the contribution statements (Lines 99-107, highlighted in blue) to emphasize VisIO as our primary methodological contribution. We quote the revised contribution statements below.

1) Our primary methodological contribution is VisIO, a method for explicitly improving VLMs' visual reliance. It automatically learns adaptive visual intervention parameters, in contrast to conventional methods that rely on manually specified intervention strengths.

2) VisIO does not require labeled VQA data for optimization. It constructs its own supervision through answer-flipping visual counterfactual pairs using only a small set of unlabeled target videos, and effectively selects pairs with high intervention potential for optimization.

3) We introduce VIS, a model- and sample-specific metric for measuring visual sensitivity. It can identify potential visual grounding failures across various VQA benchmarks without requiring grounding annotations.

[3] Xiao J, et al. Can I Trust Your Answer? Visually Grounded Video Question Answering. CVPR 2024.

[4] Villa A, et al. Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models. CVPR 2025.

I thank the authors for their thorough rebuttal and additional experiments. I believe that my original concerns have been satisfactorily addressed, particularly through the cross-dataset transfer experiments, sample-level transition analysis, and invalid-prediction robustness evaluation. These additions substantially strengthen the empirical support for the proposed approach, and I appreciate the authors efforts to clarify the novelty and scope of their contributions.

My remaining reservation is primarily about the overall strength of the contribution rather than its technical soundness. While the proposed counterfactual supervision and adaptive intervention optimization are interesting, the method builds on existing visual attention intervention techniques. Moreover, improved sensitivity to visual perturbations, although valuable, is not necessarily equivalent to demonstrating stronger causal grounding in the answer-relevant visual evidence. Thus, I maintain my weak accept rating.

Reviewer_mR4gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem of Language bias in Vision Language Models (VLMs), in two major capacities: first identification and measuring of this bias, and second by proposing a head optimization tuning of VLMs that promotes reliance on visual information to solve Visual Question Answering (VQA) tasks. The optimization process, called Adaptive Visual Intervention Optimization, creates counterfactual pairs in the training data where the same question must have different answers based on the visual information. The authors propose a system to select counterfactual constructions that maximize model adaptation, and embed this process into the optimization cycle itself, so that the counterfactual construction and selection does not require manual annotations.

Critical strengths

Critical Strength • The paper is clear and well written. • The method description is detailed and well justified, supporting reproducibility. • The results and discussion are convincing of the benefits of the proposed adaptation mechanism.

Critical weaknesses

The conclusion section of the paper is no more than a very short summary of the paper. A stronger discussion of the implication of these results and the limitations of the work would strengthen the main paper and should not be moved to the supplementary material.

Questions for authors

The identification of this language bias is presented in the paper as one of the main contributions of the paper. This bias is, however, already known for many years, being clearly reported in a perspective similar to this paper’s in (Goyal et al., ”Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering”, 2019). Could you clarify in what way your identification of this problem is novel? And possibly relate your work with these previous attempts to identify and tackle language bias in VQAs?

1 comment

We appreciate the reviewer's valuable comments.

Question: The novelty and distinction with previous language-bias studies.

  1. We clarify that we focus on visual insensitivity rather than language priors. Visual insensitivity is not limited to language bias and may also arise from failures in visual perception, reasoning, etc. Accordingly, our goal is to strengthen visual reliance rather than specifically mitigate language bias.

  2. We identify potential grounding failures without requiring grounding annotations. Prior language-prior methods are not designed to directly diagnose such failures [1–2], while prior evaluations rely on additional grounding annotations [3–4].

  3. More importantly, we highlight that our primary methodological novelty is VisIO. The main distinctions are:

    1. VisIO automatically learns adaptive intervention parameters, while conventional methods typically use manually specified or fixed intervention parameters [5–8].

    2. VisIO does not require labeled VQA data for optimization. It constructs its own supervision through answer-flipping visual counterfactual pairs using only a small set of unlabeled target videos, and selects pairs with high intervention potential for optimization.

    3. VisIO explicitly improves sensitivity to contrastive visual changes by jointly optimizing counterfactual pairs instead of individual examples.

These designs are specifically introduced for effective intervention-aware optimization and have not been explored in prior visual intervention methods. We have highlighted these distinctions in Related Work (Lines 117–121, 126–132, highlighted in blue) and contributions (Lines 99-106) of the updated manuscript.

[1] Goyal Y, et al. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.

[2] Agrawal A, et al. Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering. CVPR 2018.

[3] Xiao J, Yao A, Li Y, et al. Can I Trust Your Answer? Visually Grounded Video Question Answering. CVPR 2024.

[4] Villa A, Léon J, Soto A, et al. Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models. CVPR 2025.

[5] Sarkar S, et al. Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression. EMNLP 2025.

[6] He J, et al. Cracking the Code of Hallucination in LVLMs with Vision-Aware Head Divergence. ACL 2025.

[7] Su J, et al. Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention. ACL 2025.

[8] Wu J, et al. SHARP: Steering Hallucination in LVLMs via Representation Engineering. EMNLP 2025.

Weakness: The conclusion section is no more than a very short summary of the paper.

Thank you for your helpful suggestion. We have reorganized the conclusion and incorporated the discussion and limitations into it in the revised manuscript (Lines 486-502, highlighted in blue). We provide the updated conclusion below for convenience.

VLMs can produce correct yet visually insensitive predictions, which are often obscured by conventional accuracy. They deserve particular attention because their correctness may not be grounded in visual evidence. We therefore introduce VIS, a model- and sample-specific metric that quantifies visual sensitivity without requiring grounding annotations. This makes VIS directly applicable across existing VQA benchmarks and indicates potential grounding failure.

More importantly, we propose VisIO to strengthen visual reliance through adaptive intervention. It self-constructs supervision and selects pairs with high intervention potential for optimization. VisIO optimizes only hundreds of lightweight parameters using only a small set of unlabeled videos, offering an efficient and annotation-free adaptation scheme. Across multiple VLMs and VQA benchmarks, it consistently improves accuracy while reducing visual insensitivity.

These results suggest that strengthening visual reliance does not necessarily require large-scale retraining or human-annotated visual supervision, and lightweight and adaptive intervention can provide an effective alternative. However, the current counterfactual construction is limited to a predefined set of automatically verifiable manipulations and question templates. In addition, VisIO introduces additional pre-inference computation for counterfactual construction, pair selection, and intervention optimization. Extending VisIO to more diverse counterfactual supervision and more efficient adaptation remains an important direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.