acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Visual Ignorance in Vision-Language Models

Abstract

Vision-language models (VLMs) achieve high accuracy on many visual question-answering benchmarks, yet it remains unclear whether this accuracy reflects reliable use of the visual details a question depends on. We examine this tension through the lens of visual ignorance: cases where a model can represent task-relevant visual evidence without using it to determine the answer. Across three VLMs and twelve benchmarks, we track the answer-relevant information decodable at each decoder layer and measure when it becomes coupled to the final output. The visual evidence needed for the correct answer can already be read out from intermediate layers, yet it has little effect on the output until the final layers of the decoder. Causal residual interchange confirms that shifting the answer requires the late states, not the earlier ones. We then evaluate all twelve benchmarks under progressive Gaussian blur and find that substantial fractions of predictions survive the entire degradation sequence while continuing to receive benchmark credit. These results distinguish the availability of visual information from its influence on prediction, and show that standard accuracy can coexist with limited sensitivity to the tested visual detail. They motivate training and evaluation that make answer-relevant visual distinctions necessary to answer correctly.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, visual grounding, language priors, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_obHAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper works on whether VLMs really utilize visual details to answer questions. The authors define visual ignorance: cases where a model can represent task-relevant visual evidence without using it to determine the answer. And they study on visual ignorance via two lenses.

Lens 1. The authors do probing on each layer, showing the answer-relevant visual information is decodable at intermediate layers but strongly coupled to output changes only late. Besides, replacing a late state shifts the answer margin.

Lens 2. They then evaluate all twelve benchmarks under progressive Gaussian blur and find that substantial fractions of predictions survive the entire degradation sequence while continuing to receive benchmark credit.

Last, the authors discuss how these results motivate training and evaluation that make answer-relevant visual distinctions necessary to answer correctly.

Critical strengths
  • The authors separate the two steps of VLMs: (1) collecting visual information from the images; (2) use the information for answering. The probing experiment show the existing models are actually good at (1) but not in (2).

  • The paper is well-written and easy to understand.

Critical weaknesses
  • Lens 2. My largest concern is about the second lens.

    • First, this evaluation method seems not very relevant to the previous context. The authors study on visual ignorance at the beginning, where a model can represent task-relevant visual evidence but do not use it to determine the answer. However, in the Gaussian-blur evaluation, the images are blurred and the model may neither represent visual evidence nor use it for answering. Thus, I'm not sure how the Guassian blur method is related to the visual ignorance. It looks like a general method to test the VLMs' robustness to visual inputs.
    • Second, the evaluation in lens 2 is not very novel. Prior works discuss a few ways to test VLMs, such as replacing, editing, removing, and deteriorating images or videos in benchmarks.
    • Last, after a mild Gaussian blur, I think some images are still informative to answer the questions. Thus, this evaluation seems to actually test the VLM robustness to visual evidence instead of ignorance. Why not just remove the images or replace them with all-zero tensors?
  • Implications. Previous works already show that VLMs and benchmarks need to pay more attention to visual evidence. Visual ignorance is interesting because it is a special case where a model actually capture the task-relevant visual evidence. It tells us that capturing and utilizing visual evidence are actually two different things, but unfortunately it doesn't give us more insights. The final discussion is still that "we need to pay more attention to visual evidence". The authors are recommended to provide deeper exploration on the implications of visual ignorance. For example, what algorithms or what special solutions can we design if we know the visual ignorance issue?

Questions for authors
  • How is the lens 2 related to visual ignorance?
  • What does visual ignorance motivate VLM training and evaluation?
6 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

We thank the reviewer for their valuable feedback.


W1. First, this evaluation method seems not very relevant to the previous context. The authors study on visual ignorance at the beginning, where a model can represent task-relevant visual evidence but do not use it to determine the answer. However, in the Gaussian-blur evaluation, the images are blurred and the model may neither represent visual evidence nor use it for answering. Thus, I'm not sure how the Guassian blur method is related to the visual ignorance. It looks like a general method to test the VLMs' robustness to visual inputs.

A1. We apologize for not making the connection clear. An unchanged answer under mild blur can reflect robustness when sufficient answer-relevant information remains. Our evaluation tracks predictions through the entire degradation sequence and measures how much benchmark credit is retained by answers that remain unchanged as visual detail is progressively removed. We agree that this behavioral analysis does not independently identify encoded-but-unused information in each example.

The two analyses address the same main problem: whether answer-relevant visual evidence is effectively used in answering. The internal analysis shows that decodable answer-relevant information need not produce the appropriate answer update. The benchmark analysis examines the evaluation consequence: substantial benchmark credit can persist even when predictions remain unchanged throughout progressive visual degradation. Appendix J further reports that more than 40% of answers remain unchanged after image removal for each of the three models, showing that many predictions persist even without image input. Together, these analyses connect a limitation in the use of represented information to the ability of benchmarks to reward visually insensitive predictions. We will clarify this connection in the revised manuscript.


W2. Second, the evaluation in lens 2 is not very novel. Prior works discuss a few ways to test VLMs, such as replacing, editing, removing, and deteriorating images or videos in benchmarks.

A2. We agree that image corruption alone is not novel when evaluating VLMs. However, as discussed in Section 2, we mainly focus on the portion of the models' answers that remain unchanged throughout multiple blur levels. This design has two benefits: (1) Progressive Gaussian blur provides a sequence of increasingly degraded inputs, allowing us to track whether predictions respond as visual detail is removed; (2) Consecutive agreement requires the prediction to match the original answer at every degradation level, excluding cases where it changes and later returns to that answer. Our contribution is a progressive evaluation protocol that measures both the prevalence of predictions that remain unchanged throughout visual degradation and the benchmark credit these predictions retain across twelve benchmarks.


W3. Last, after a mild Gaussian blur, I think some images are still informative to answer the questions. Thus, this evaluation seems to actually test the VLM robustness to visual evidence instead of ignorance. Why not just remove the images or replace them with all-zero tensors?

A3. We apologize for not explicitly mentioning other evaluation techniques in the main text. In Appendix J, specifically in Appendix Table 2, we present other evaluation techniques, including image removal. We observe that the averaged proportion of invariant answers before/after image removal is over 40% for the 3 models, which supports the results of progressive blur. Image removal tests whether predictions persist in the complete absence of image input. Progressive blur additionally tracks whether an answer remains unchanged at every stage of visual degradation. Agreement between the original-image and text-only answers alone cannot distinguish sustained invariance from a prediction that changes during degradation and later returns to the original answer. We therefore use image removal as a complementary control for the progressive evaluation. In the revised manuscript, we will mention the text-only evaluation explicitly in the main text.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

W4. Implications. Previous works already show that VLMs and benchmarks need to pay more attention to visual evidence. Visual ignorance ... tells us that capturing and utilizing visual evidence are actually two different things, but unfortunately it doesn't give us more insights. The final discussion is still that "we need to pay more attention to visual evidence". The authors are recommended to provide deeper exploration on the implications of visual ignorance. For example, what algorithms or what special solutions can we design if we know the visual ignorance issue?

A4. We thank the reviewer for this suggestion. The Discussion already distinguishes whether visual information is available from whether it actually affects the output, and it gives concrete suggestions for training and evaluation. We agree that the paper does not yet provide a concrete solution. Below we provide a possible direction and discuss its implications.

A useful way to frame visual ignorance is through shortcut learning [1]: the tendency of deep networks to learn decision rules that work on the training distribution but fail when the test distribution requires information the shortcut bypasses. Because shortcuts can survive standard in-distribution evaluation, high benchmark accuracy alone may not reveal their use. Visual ignorance has exactly this structure. The model can answer from question-answer priors or category-level expectations, which are sufficient for much of the training data, and the decisive visual detail is never forced to influence the output. Our progressive blur and text-only evaluations reduce or remove visual evidence while retaining possible shortcuts based on the question and learned language priors. The high invariance rates are consistent with continued reliance on these shortcuts even when visual evidence is reduced or absent. This is consistent with VQA models relying on question–answer priors [2], contrastive vision–language models doing well on retrieval without binding objects to attributes and relations [3], and memorized world knowledge competing with conflicting visual input in modern VLMs [4]. We treat this as a hypothesis about where the bias comes from, not as a proven general rule for every model.

This framing makes a testable prediction: if the shortcut is the problem, then supervision that makes the shortcut insufficient should reduce visual ignorance. We tested this hypothesis with the VisOnlyQA dataset [5]. Its labels depend on precise visual attributes and relations: small changes in geometry or spatial layout can change the answer, so category-level priors are not enough. We fine-tuned the full Qwen3.5-9B model for 300 optimization steps with an effective batch size of 18, then repeated the causal experiment on the same VLMBias normal-counterfactual pairs. The intervention effect grows in the mid-to-late layers:

Mean causal effectBaseVisOnlyQA
Layers 16–230.3040.429
Layers 24–310.7140.832

The stronger mid-to-late effects come with a substantial gain on VLMBias and a non-trivial gain on HallusionBench:

BenchmarkBaseVisOnlyQA
VLMBias accuracy14.83%24.93%
HallusionBench question accuracy47.91%51.21%

These results support training with supervision that depends on precise visual attributes. The model responds more strongly to inserted counterfactual signals in the mid-to-late layers and improves on both VLMBias and HallusionBench. Together, these results demonstrate that training on more fine-grained supervision signals may help alleviate the visual ignorance problem.

At inference time, the same logic gives a simple diagnostic signal. If a model's answer stays invariant after a controlled corruption of the image, the prediction may rely on memorized priors rather than the detail the question asks about. Practitioners can use this signal to flag unreliable predictions before acting on them.

More broadly, visual ignorance is not a single-dataset problem. It is a research problem that spans data, objectives, and evaluation: training signals that depend on the key visual detail, objectives that reward sensitivity to that detail, and metrics that distinguish real use of evidence from coincidental correctness. We see these results as one step in that direction; much work remains across tasks, architectures, and model scales.

[1] Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence 2020.

[2] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.

[3] When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It? ICLR 2023.

[4] Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts. EMNLP 2025.

[5] VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information. COLM 2025.

I thank the authors for the detailed rebuttal. However, some of my concerns remain.

W1. The evaluation method seems not very relevant to the previous context (as also noted by Reviewer YPAV).

Rebuttal: "The two analyses address the same main problem: whether answer-relevant visual evidence is effectively used in answering."

What I expected is that: the paper does some analysis first and then propose a new evaluation based on the analysisi. Here, the second part (the evaluation) is not very dependent on the first part (the analysis) and they look like two different analyses. But I think I can accepet this, and in this case, the paper just proposes two things for the same problem.

W2 & 3. The evaluation in lens 2 is not very novel and after a mild Gaussian blur, I think some images are still informative to answer the questions (as also noted by Reviewer 7qMu).

Rebuttal: The authors mainly focus on the portion of the models' answers that remain unchanged and in the appendix, present other evaluation techniques, including image removal.

My concern is that the Gaussian blur evaluation doesn't look better than previous methods. For visual ignorance, can we just use image removal or use some blank images to test LVLMs? It seems that the Gaussian blur is indeed a different method but not better than these simple ways.

W4. I think there might be some misunderstanding in my comment. I was not saying that the paper doesn't give concrete suggestions for training and evaluation. The analyses that the paper proposes are interesting but they do not provide something new, something different from previous works. The suggestions are not specific to visual ignorance, but general to LVLMs.

As I comment in the review, "Previous works already show that VLMs and benchmarks need to pay more attention to visual evidence. Visual ignorance is interesting because it is a special case where a model actually capture the task-relevant visual evidence. It tells us that capturing and utilizing visual evidence are actually two different things, but unfortunately it doesn't give us more insights."

I appreciate the authors for the new experiment and the detailed response. I think the topic is overall interesting and the authors could rethink about the lens 2 and go deeper into it.

Reviewer_YPAVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the visual ignorance in Vision-Language Models (VLMs) through the lens of internal answer decodability and degradation-based evaluation. Starting with paired normal/counterfactual images, the first analysis probes the decodability across model layers and replaces latent states to compare answer shifts. The results show that answer-related evidence is often decodable from intermediate layers. To further quantify the visual insensitivity of VLMs, a progressive blur evaluation is applied to three VLMs on twelve datasets. These evaluations highlight the importance of visual evidence for model evaluations and training.

Critical strengths

Through controlled analysis under a counterfactual setting and blur-based evaluation, the paper shows that VLMs may not fully exploit available visual evidence when answering questions. This provides insights into model evaluation and training.

Critical weaknesses
  • The finding of decodability in a counterfactual setting is not completely new. Prior works [1,2] have already applied probing to VLMs and show that visual concepts/attributes are linearly decodable. The authors should compare with previous methods and properly position the paper in the literature.
  • Limited analytical depth on degradation-based evaluation. First, the input image is barely visible under large Gaussian blur kernels (see Fig. 6). The performance degradation is therefore expected. In this case, it is suggested to compare with text-only input to justify how VLMs rely on text and image to answer the question. Second, different tasks may require different visual evidence. Reporting a single aggregated accuracy across subtasks in the dataset does not fully reflect the visual insensitivity of VLMs. A more comprehensive evaluation is recommended.

Reference

[1] Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving. TMLR 2026.

[2] Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts. COLM 2026.

Questions for authors
  • The connection between the two analysis sections is somewhat weak. The first analysis section investigates the decodability, but the second one focuses on visual insensitivity. The internal connections between these sections are not well established.
  • Limited model evaluations. The paper only evaluates three models, including Qwen2.5-VL-3B/7B and LLaVA-v1.6-Mistral-7B. It is unclear whether the findings can generalize to other model families or larger models.
10 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/3)

We thank the reviewer for their valuable feedback.


W1. The finding of decodability in a counterfactual setting is not completely new. Prior works [1,2] have already applied probing to VLMs and show that visual concepts/attributes are linearly decodable. The authors should compare with previous methods and properly position the paper in the literature.

A1. We thank the reviewer for pointing out these relevant works. We agree that visual decodability, including cases where a model encodes relevant information but answers incorrectly, has been demonstrated previously. Section 3 builds on this finding by examining when image-induced internal changes become coupled to output changes and where their causal influence increases sharply.

Regarding [1], the authors characterize how visual concepts are represented and show that successful decoding does not guarantee a correct answer. Their steering experiments further validate that the learned concept directions can influence generation. However, they do not trace how the causal influence of image-induced state differences changes across decoder depth. Our analysis addresses this question, revealing a sharp late transition and localizing its largest increase to an attention step.

Regarding [2], the authors establish an encoding–grounding dissociation and use activation interventions to investigate how VLMs resolve conflicts between visual attributes and prior knowledge. However, their study is confined to color and relative-size conflicts on Visual CounterFact. The dataset deliberately reduces visual confounds through white backgrounds, clearly distinguishable color changes, and object arrangements that minimize depth ambiguity. Moreover, its causal patching and steering evaluations cover only the Color split. These controlled settings are valuable for isolating attribute conflicts, but leave open how the findings extend to more varied visual contexts, where recognizing familiar content can coexist with overlooking the particular details needed to answer the question. Our internal analysis extends the investigation to this setting and examines how changes in visual evidence propagate from intermediate representations to the final prediction. Beyond establishing that relevant information is decodable, we connect image-induced internal changes to their output consequences and identify a sharp late increase in causal influence. In Qwen2.5-VL-3B, we further localize this increase to an attention step. This adds a concrete account of where the influence of visual information changes during decoding, extending the evidence beyond the existence of an encoding–answering gap and the attribute conflicts studied in [2].

Neither work examines how standard benchmarks reward predictions that remain insensitive to visual degradation. Our second analysis addresses this evaluation question across twelve benchmarks, measuring both the prevalence of answers that remain unchanged throughout the blur sequence and the accuracy those answers retain.

We will revise the related-work discussion to make these extensions explicit while acknowledging that decodability and the encoding–answering gap have established precedents.

AuthorsOpenReview ↗

Authors' Rebuttal (2/3)

W2.1. Limited analytical depth on degradation-based evaluation. First, the input image is barely visible under large Gaussian blur kernels (see Fig. 6). The performance degradation is therefore expected. In this case, it is suggested to compare with text-only input to justify how VLMs rely on text and image to answer the question.

A2.1. We apologize for not explicitly mentioning text-only experiments in the main text. In Appendix J, we conduct experiments on more image corruption techniques. Appendix Table 2 reports agreement with the clean-image answer under the strongest-blur and text-only conditions. The averaged portion of invariant answers before and after image removal is over 40% for all models tested, and is closely related to the portion under progressive blur settings. However, the answer agreement before and after image removal may be due to coincidence when the answer space is discrete and limited (e.g. in multi-choice question benchmarks), so we believe that the consecutive agreement across all blur levels is more convincing. We also argue that, even when the most blurred images can provide vague cues for the questions, the model is still neglecting the fact that the image is severely corrupted but directly answers the questions based on the unclear visual evidence left, which is a sign of visual ignorance. In the revised manuscript, we will clearly describe the text-only comparisons in the main text.


W2.2. Second, different tasks may require different visual evidence. Reporting a single aggregated accuracy across subtasks in the dataset does not fully reflect the visual insensitivity of VLMs. A more comprehensive evaluation is recommended.

A2.2. We thank the reviewer for this helpful advice. We agree that a single aggregate may obscure heterogeneity in the model's response to visual degradation. In the revision, we will report clean accuracy, strongest-blur accuracy, full-path consecutive agreement, invariant-subset accuracy, prediction distributions, and invalid-output counts for every benchmark-native subtask. Below is a key finding drawn from subset-aware analyses. Metrics are averaged across models evaluated.

Fine-grained visual evidence is more blur-sensitive than coarse perception. On MMStar, all six categories have the same number of examples, making the contrast particularly transparent:

MMStar categoryClean AccuracyStrongest Blur AccuracyConsecutive agreement
Coarse perception72.4%34.4%21.2%
Math71.6%31.2%15.6%
Instance reasoning75.6%24.0%12.8%
Fine-grained perception62.8%18.8%8.8%
Science & Technology55.2%18.0%7.6%
Logical reasoning78.0%21.2%6.8%

As shown in the last column, the observed phenomenon of consecutive invariance is weaker when the task demands fine visual discrimination or complex visual reasoning. We hypothesize that this reflects the heterogeneity of the portions of coarse and fine supervision in the training data: coarse-grained supervision signals like holistic perception are easier to collect and more prevalent in real life, while fine-grained supervision is scarcer. Therefore, the model may fall back to the familiar answers more often on coarse perception tasks. However, we acknowledge that more rigorous research should be done to unveil the core causal relationships around this phenomenon. We will add the complete benchmark-native subtask tables and the respective analyses in the revised manuscript.


Q1. The connection between the two analysis sections is somewhat weak. The first analysis section investigates the decodability, but the second one focuses on visual insensitivity. The internal connections between these sections are not well established.

AQ1. We thank the reviewer for pointing out that the connection between the two analyses was not sufficiently explained. The two analyses address the same main problem: whether answer-relevant visual evidence is effectively used in answering. The internal analysis shows that decodable answer-relevant information need not produce the appropriate answer update. We examine this gap using matched normal–counterfactual image pairs. The benchmark analysis examines the evaluation consequence: substantial benchmark credit can persist even when predictions remain unchanged throughout progressive visual degradation. We will emphasize the connection between the two analyses in the revised manuscript.

AuthorsOpenReview ↗

Authors' Rebuttal (3/3)

Q2. Limited model evaluations. The paper only evaluates three models, including Qwen2.5-VL-3B/7B and LLaVA-v1.6-Mistral-7B. It is unclear whether the findings can generalize to other model families or larger models.

AQ2. We agree that causal evidence beyond the originally evaluated architectures is necessary. We would like to first clarify that Section 3.7 already reports observational internal-output coupling in Qwen2.5-VL-7B and LLaVA-v1.6-Mistral-7B. We now add causal evidence from two further architectures, Qwen3.5-9B and Gemma4-12B, as well as external progressive-blur evidence on both models.

Internal causal analysis.

Qwen3.5-9B. Qwen3.5-9B has 32 language blocks, interleaving 24 Gated DeltaNet blocks with 8 full-attention blocks. On the common 194-pair set, we replace the normal run's final prompt-token state at each block output with its counterfactual counterpart, and measure the induced logit-margin update Δt averaged over the two question wordings. Because the architecture interleaves linear and full attention, the Qwen2.5 attention/MLP decomposition does not directly apply; for this model we therefore report block-output localization only.

Gemma4-12B. Gemma4-12B is a frozen dense 48-layer model. We apply the same block-output intervention to the same 194 pairs and average over the two wordings. In addition, we perform boundary localization at all 48 layers, replacing the final prompt-token state at the layer input, the post-attention residual, and the block output, which yields attention and MLP gains within each layer.

ModelNum. LayersFirst QuarterSecondThirdFourthFinal Layer
Qwen3.5-9B320.0540.0900.3040.7141.612
Gemma4-12B48-0.003-0.0040.0390.2050.281

Analysis. The table reports mean Δt over equal-depth quarters, averaged over the two question wordings. Both models share the same depth profile: early quarters are near zero or small, and the effect rises across the third and fourth quarters, with the largest increase in the final quarter. In Qwen3.5, the quarters rise from 0.054 in the first to 0.714 in the fourth and reach 1.612 at the final block; in Gemma, the final quarter is 0.205 and the final block 0.281. For Qwen3.5-9B, the largest single-layer increase occurs between layers 29 and 30, where the pair-averaged Δt rises from 0.609 to 1.200 (+0.591). Gemma's boundary localization places the transition at layer 34: attention gain is +0.072, compared with -0.022 at layer 33; MLP gain is -0.0003. This reproduces the attention-mediated, non-MLP pattern of Qwen2.5-VL-3B at layer 27. Together, Qwen3.5 shows that the causal concentration survives a hybrid linear/full-attention architecture, and Gemma shows that the attention-level localization is not specific to Qwen2.5-VL-3B.

External behavioral analysis.

Both new architectures also exhibit blur-insensitive behavior in benchmark evaluation. The table reports the unweighted mean over the 12 benchmark units used in the evaluation. For each benchmark, we first compute the model-level rate and then average across benchmarks; this is a descriptive cross-benchmark summary rather than a pooled example-level metric.

ModelBenchmark unitsMean clean acc.Mean strongest-blur acc.Mean consecutive agreement
Qwen3.5-9B1258.88%36.07%25.07%
Gemma4-12B1248.39%33.28%30.42%

Interpretation. Both models retain a non-trivial fraction of unchanged predictions over the complete progressive-blur path, despite a substantial mean drop from clean to strongest-blur accuracy. These external results are behavioral rather than causal: they establish that blur-insensitive predictions persist in both a newer hybrid Qwen architecture and a larger, distinct Gemma family.

Conclusion. Therefore, the late-layer mechanism and the consecutive agreement phenomenon are not confined to the models originally evaluated in our paper. Qwen3.5-9B establishes causal late-layer concentration in a hybrid linear/full-attention architecture, while Gemma4-12B establishes the same late concentration and an attention-mediated, non-MLP transition in a larger, different model family. The blur analysis shows that the problem of visual ignorance still manifests in both models. We will include these results in the revised manuscript.

Reviewer_7qMuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether VLMs genuinely use task-relevant visual evidence or rely on learned language priors to generate answers. Across three VLMs and twelve benchmarks, the authors combine layer-wise probing, causal interventions, and progressive image blurring. They find that answer-relevant information can be decoded from intermediate layers but often influences the final prediction only in later layers. Their experiments also show that many predictions remain unchanged under severe image degradation while retaining high benchmark accuracy. The findings highlight a gap between a model’s ability to represent visual information and its actual use of that information.

Critical strengths
  1. The paper addresses a fundamental limitation of VLM evaluation, where high accuracy does not necessarily indicate genuine visual understanding.
  2. Combines representation probing, matched counterfactual image pairs, and causal residual interchange to investigate both the availability and influence of visual evidence.
  3. Studies three VLMs across twelve benchmarks, providing evidence that visual insensitivity is not restricted to a single model or dataset.
  4. The consecutive-agreement criterion under progressive blurring offers a more informative measure of prediction persistence than comparing only the original and most heavily blurred images.
  5. The results provide useful guidance for designing training data and evaluation benchmarks that require models to rely on decisive visual details.
Critical weaknesses
  1. The causal residual-interchange experiments and attention-level localization are conducted only on Qwen2.5-VL-3B. The evidence for the other architectures is observational, leaving the generality of the proposed causal mechanism uncertain.
  2. An unchanged answer under blurring does not necessarily indicate visual ignorance. Some questions may remain answerable from residual visual cues, scene context, or information preserved at intermediate blur levels. The metric identifies insensitivity to degradation but does not independently establish that the model ignored the relevant evidence in every case.
  3. The internal investigation focuses primarily on numerical answers from a particular counterfactual benchmark. It remains unclear whether the same late-layer coupling pattern holds for more diverse visual reasoning tasks, such as spatial relations, text recognition, and complex diagrams.
  4. The observed late-layer coupling and attention-mediated transition are compelling, but the paper does not establish why this transition occurs or demonstrate that changing it would systematically improve visual reasoning across tasks.
Questions for authors

Please see the mentioned weaknesses.

4 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/3)

We thank the reviewer for their valuable feedback.


W1. The causal residual-interchange experiments and attention-level localization are conducted only on Qwen2.5-VL-3B. The evidence for the other architectures is observational, leaving the generality of the proposed causal mechanism uncertain.

A1. We thank the reviewer for this suggestion. We now extend the causal analysis to Qwen3.5-9B and Gemma4-12B.

Qwen3.5-9B. Qwen3.5-9B has 32 language blocks, interleaving 24 Gated DeltaNet blocks with 8 full-attention blocks. On the common 194-pair set, we replace the normal run's final prompt-token state at each block output with its counterfactual counterpart, and measure the induced logit-margin update Δt averaged over the two question wordings. Because the architecture interleaves linear and full attention, the Qwen2.5 attention/MLP decomposition does not directly apply; for this model we therefore report block-output localization only.

Gemma4-12B. Gemma4-12B is a frozen dense 48-layer model. We apply the same block-output intervention to the same 194 pairs and average over the two wordings. In addition, we perform boundary localization at all 48 layers, replacing the final prompt-token state at the layer input, the post-attention residual, and the block output, which yields attention and MLP gains within each layer.

ModelNum. LayersFirst QuarterSecondThirdFourthFinal Layer
Qwen3.5-9B320.0540.0900.3040.7141.612
Gemma4-12B48-0.003-0.0040.0390.2050.281

Analysis. The table reports mean Δt over equal-depth quarters, averaged over the two question wordings. Both models share the same depth profile: early quarters are near zero or small, and the effect rises across the third and fourth quarters, with the largest increase in the final quarter. In Qwen3.5, the quarters rise from 0.054 in the first to 0.714 in the fourth and reach 1.612 at the final block; in Gemma, the final quarter is 0.205 and the final block 0.281. For Qwen3.5-9B, the largest single-layer increase occurs between layers 29 and 30, where the pair-averaged Δt rises from 0.609 to 1.200 (+0.591). Gemma's boundary localization places the transition at layer 34: attention gain is +0.072, compared with -0.022 at layer 33; MLP gain is -0.0003. This reproduces the attention-mediated, non-MLP pattern of Qwen2.5-VL-3B at layer 27. Together, Qwen3.5 shows that the causal concentration survives a hybrid linear/full-attention architecture, and Gemma shows that the attention-level localization is not specific to Qwen2.5-VL-3B.

Therefore, the late-layer mechanism is not confined to the models being evaluated in our paper. We will include these results in the revised manuscript.


W2. An unchanged answer under blurring does not necessarily indicate visual ignorance. Some questions may remain answerable from residual visual cues, scene context, or information preserved at intermediate blur levels. The metric identifies insensitivity to degradation but does not independently establish that the model ignored the relevant evidence in every case.

A2. We thank the reviewer for pointing out that some visual information may still remain in the image after blurring. We would like to first clarify that each intermediate blur level alone does not support visual ignorance. We recognize visual ignorance mainly by two mechanisms: (1) the consistent agreement of answers at all blur levels, which significantly rules out the chance of coincidences; (2) the fact that the model's answer remains unchanged even at the strongest blur level, where as Figure 6 shows, no object outlines can be recognized anymore. In this case, the model is ignoring the fact that the image is severely corrupted and still outputs the answer as if it is not. A responsible AI assistant should report the obvious situation that the image can barely be read, which may be due to the user uploading a wrong or irrelevant image, rather than directly answering the question from the scarce and obscure visual information left. Also, in Appendix J, we report other corruption techniques, including text-only, where the image is completely removed. According to Appendix Table 2, the averaged portion of invariant answers before and after image removal is over 40% for all models tested, which demonstrates that residual visual information alone does not explain the high rate of consecutively invariant answers in progressive blur settings. Overall accuracy exceeds 30% for every model under both the strongest-blur and text-only conditions. These findings show that many predictions persist without image input, consistent with substantial reliance on learned language priors. This observation is consistent with the fact that the models perform poorly on counterfactual datasets such as VLMs Are Biased. In the revised manuscript, we will clearly describe the text-only experiments in the main text.

AuthorsOpenReview ↗

Authors' Rebuttal (2/3)

W3. The internal investigation focuses primarily on numerical answers from a particular counterfactual benchmark. It remains unclear whether the same late-layer coupling pattern holds for more diverse visual reasoning tasks, such as spatial relations, text recognition, and complex diagrams.

A3. We thank the reviewer for this important point. We agree that the original internal analysis is centered on numerical VLMBias pairs. To test whether the late-layer concentration of causal influence is specific to numerical counting, we extend the causal residual interchange to three additional visual-reasoning families in Qwen2.5-VL-3B: spatial relations (CounterCurate), text recognition (OCR questions in HallusionBench), and complex figures and diagrams (the figure and diagram subset of HallusionBench). For each task we follow the residual-interchange protocol of Section 3.5: we either replace only the normal run's final prompt-token state at the output of a single decoder block with its counterfactual counterpart (the last two columns of the table, for layers 27 and 35), or replace this state simultaneously across a contiguous layer range (the four range columns). We use the output margin from Section 3.3, defined as the logit of the edited target answer minus the logit of the source reference answer, with both labels held fixed across the two image conditions. The induced logit-margin update is Δt; positive values indicate movement toward the edited target.

Task4–1012–1825–3132–35L27L35
Spatial relations (CounterCurate left/right)-0.001-0.003+0.647+0.685+0.501+0.685
Text recognition (HallusionBench OCR)-0.016-0.013+4.048+4.156+3.527+4.156
Complex diagrams (HallusionBench figure/diagram)+0.008+0.003+2.127+2.161+1.835+2.161
HallusionBench VD (all feature types)-0.000+0.002+1.478+1.513+1.276+1.513

We make two observations. First, in every task the mean effects in the early and middle ranges are close to zero, whereas the late ranges are strongly positive. Second, the layerwise intervention results show the same abrupt rise between layers 24 and 28 rather than a gradual increase. These results show that late causal concentration extends beyond numerical counterfactuals to spatial, OCR, and diagram tasks. We will add these results and the corresponding protocols to the revised manuscript.

AuthorsOpenReview ↗

Authors' Rebuttal (3/3)

W4. The observed late-layer coupling and attention-mediated transition are compelling, but the paper does not establish why this transition occurs or demonstrate that changing it would systematically improve visual reasoning across tasks.

A4. We thank the reviewer for highlighting the need to explain the transition and examine its practical implications. The problem extends beyond visual influence becoming pronounced only in late layers: even when the counterfactual signal reaches the output, its contribution can remain too weak to overturn the familiar answer.

Localizing this transition does not imply that updating a small collection of modules will remove the underlying bias. Familiar-answer preferences may reflect patterns in the pre-training distribution and be reinforced across multiple components. One plausible source is supervision that permits shortcuts: VQA models can exploit question-answer priors [1], and contrastive vision-language models can perform well on retrieval without correctly binding objects to their attributes and relations [2]. Memorized world knowledge also competes with conflicting visual input in modern VLMs [3]. We treat this explanation as a hypothesis about the origin of the bias.

We therefore test broader adaptation through full-model fine-tuning on VisOnlyQA [4]. Its supervision is tied to precise visual attributes and relations: small differences in the depicted geometry or spatial configuration can determine the final label. Correct predictions therefore require distinguishing details that category-level expectations alone cannot determine.

After training Qwen3.5-9B for 300 optimizer updates with an effective batch size of 18, we repeated the causal experiment on the model with the same VLMBias normal-counterfactual pairs. The values below show that the trained model is more responsive to the inserted counterfactual signals.

Mean causal effectBaseVisOnlyQA
Layers 16–230.3040.429
Layers 24–310.7140.832

The stronger middle-to-late effects accompany improvements on two counterfactual benchmarks, VLMBias and HallusionBench:

BenchmarkBaseVisOnlyQA
VLMBias accuracy14.83%24.93%
HallusionBench question accuracy47.91%51.21%

These findings support this direction, although they do not establish that the benchmark gains are caused specifically by the localized transition. The underlying problem remains complex: models must override familiar priors when reliable visual evidence contradicts them while retaining useful priors when appropriate. Substantial work remains to achieve this balance across tasks and models. We consider vision-only supervision in which fine visual details determine precise labels a viable direction toward that goal.

[1] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.

[2] When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It? ICLR 2023.

[3] Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts. EMNLP 2025.

[4] VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information. COLM 2025.

Reviewer_7qMuOpenReview ↗

Response to the Comments

​Thank you for the comments and for conducting the additional experiments. I have reviewed them. I increased my score accordingly.

Comments on the paper

2 comments
Area_Chair_pDe7OpenReview ↗

Discussion phase will end on Nov 18

This is a kind reminder that discussion phase will end on Nov. 18.

Dear Reviewer YPAV and obHA, thank you for your active engagement in the discussion. Could you please check the most recent response from the authors as well?

Dear Reviewer 7qMu, could you please check authors' rebuttal and update your thoughts? Your early feedback would make the discussion more smooth and informative.

Thank you,

AC

AuthorsOpenReview ↗

General Response

Dear Reviewers, AC, SAC, and PC,

Thank you for reviewing and discussing our paper. Our point-by-point responses provide full details, and we summarize the main results and revisions below.


Paper Strengths

[Strengths]: The reviewers highlight the fundamental evaluation problem addressed by our paper: high accuracies on benchmarks need not indicate genuine visual understanding of VLMs (7qMu). They value our combination of probing, matched counterfactual pairs, and causal residual interchange for distinguishing the availability of visual evidence from its influence on answers (7qMu, obHA), as well as the broad evaluation, consecutive-agreement criterion, and implications for training and evaluation (7qMu, YPAV). Reviewer obHA also finds the paper well written and easy to understand.


Additional Experiments and Analyses

  • [Generalization Across Architectures] (7qMu W1, YPAV Q2): Residual interchange on Qwen3.5-9B and Gemma4-12B shows little early-layer causal influence and a pronounced late-layer rise. From the first quarter to the final block, mean output-margin updates rise from 0.054 to 1.612 in Qwen3.5 and from -0.003 to 0.281 in Gemma. Gemma's largest attention gain occurs at layer 34 (+0.072), while its MLP gain is -0.0003. The late concentration and attention-mediated pattern are therefore not specific to Qwen2.5-VL-3B.

  • [Generalization Across Tasks] (7qMu W3): In Qwen2.5-VL-3B, we extend residual interchange from numerical pairs to spatial relations, OCR, and complex diagrams. Every task shows near-zero early and middle effects, strongly positive late effects, and an abrupt rise between layers 24 and 28.

  • [Behavioral Controls] (7qMu W2, YPAV W2.1, obHA W2-3): Most samples that satisfy full-path blur invariance also satisfy clean-to-text-only agreement, but the reverse does not hold. Under the additional filter of satisfying both conditions, invariant-subset accuracy changes by less than one percentage point for every model compared to only satisfying blur invariance. At 3x3, 5x5, and 7x7 blur, overall accuracy remains above text-only accuracy on 11, 10, and 10 benchmarks, respectively. Thus, the result persists under text-only controls and at stages where visual information remains useful.

  • [Task-Level and New-Model Results] (YPAV W2.2, Q2): We add task-level results and evaluate both new models across twelve benchmark units. Strict consecutive agreement is 25.0% for Qwen3.5 and 28.4% for Gemma, while mean invariant-subset accuracy is 72.2% and 58.0%. The Gemma accuracy averages the eleven units with a nonempty strict invariant subset.

  • [Adaptation] (7qMu W4, obHA W4): Fine-tuning Qwen3.5-9B on VisOnlyQA strengthens the causal effect in layers 16-23 from 0.304 to 0.429 and in layers 24-31 from 0.714 to 0.832. Accuracy rises from 14.83% to 24.93% on VLMBias and from 47.91% to 51.21% on HallusionBench, supporting supervision whose answers depend on precise visual attributes and relations.


Clarifications

  • [Two Analyses] (YPAV Q1, obHA W1): The internal analysis asks whether image-induced state changes influence the prediction; the behavioral analysis asks whether benchmarks reward predictions insensitive to removed visual detail. They provide complementary internal and evaluation-level evidence without assuming the same mechanism in every sample.

  • [Language Priors] (YPAV W2.1): Language-prior reliance and visual ignorance are not competing explanations. When a prior determines an answer despite task-relevant visual evidence, it is a form of visual ignorance. We assess this through full-path invariance, text-only controls, image-dependent benchmarks, invariant-subset accuracy, and clear-image counterfactual interventions, not a single unchanged answer under mild blur.

  • [Prior Work] (YPAV W1): Prior work establishes decodability gaps and intervenes on visual-prior conflicts. We instead trace how matched image-induced state differences gain influence across decoder depth, identify a sharp late causal transition, and examine benchmark credit for predictions insensitive to progressive degradation.


Revision During Rebuttal

  • [Introduction and Related Work]: connection between the two analyses; positioning relative to probing and intervention studies.

  • [Sec. 3]: cross-architecture coupling and causal results for Qwen3.5-9B and Gemma4-12B.

  • [Sec. 4]: five-model behavioral evaluation; text-only and joint-invariance controls.

  • [Discussion]: implications for supervision granularity, training, and evaluation.

  • [Appendix]: cross-task residual interchange; full protocols, controls, and task-level results.

All changes are highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.