- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper works on whether VLMs really utilize visual details to answer questions. The authors define visual ignorance: cases where a model can represent task-relevant visual evidence without using it to determine the answer. And they study on visual ignorance via two lenses.
Lens 1. The authors do probing on each layer, showing the answer-relevant visual information is decodable at intermediate layers but strongly coupled to output changes only late. Besides, replacing a late state shifts the answer margin.
Lens 2. They then evaluate all twelve benchmarks under progressive Gaussian blur and find that substantial fractions of predictions survive the entire degradation sequence while continuing to receive benchmark credit.
Last, the authors discuss how these results motivate training and evaluation that make answer-relevant visual distinctions necessary to answer correctly.
Critical strengths
-
The authors separate the two steps of VLMs: (1) collecting visual information from the images; (2) use the information for answering. The probing experiment show the existing models are actually good at (1) but not in (2).
-
The paper is well-written and easy to understand.
Critical weaknesses
-
Lens 2. My largest concern is about the second lens.
- First, this evaluation method seems not very relevant to the previous context. The authors study on visual ignorance at the beginning, where a model can represent task-relevant visual evidence but do not use it to determine the answer. However, in the Gaussian-blur evaluation, the images are blurred and the model may neither represent visual evidence nor use it for answering. Thus, I'm not sure how the Guassian blur method is related to the visual ignorance. It looks like a general method to test the VLMs' robustness to visual inputs.
- Second, the evaluation in lens 2 is not very novel. Prior works discuss a few ways to test VLMs, such as replacing, editing, removing, and deteriorating images or videos in benchmarks.
- Last, after a mild Gaussian blur, I think some images are still informative to answer the questions. Thus, this evaluation seems to actually test the VLM robustness to visual evidence instead of ignorance. Why not just remove the images or replace them with all-zero tensors?
-
Implications. Previous works already show that VLMs and benchmarks need to pay more attention to visual evidence. Visual ignorance is interesting because it is a special case where a model actually capture the task-relevant visual evidence. It tells us that capturing and utilizing visual evidence are actually two different things, but unfortunately it doesn't give us more insights. The final discussion is still that "we need to pay more attention to visual evidence". The authors are recommended to provide deeper exploration on the implications of visual ignorance. For example, what algorithms or what special solutions can we design if we know the visual ignorance issue?
Questions for authors
- How is the lens 2 related to visual ignorance?
- What does visual ignorance motivate VLM training and evaluation?
6 comments
Authors' Rebuttal (2/2)
W4. Implications. Previous works already show that VLMs and benchmarks need to pay more attention to visual evidence. Visual ignorance ... tells us that capturing and utilizing visual evidence are actually two different things, but unfortunately it doesn't give us more insights. The final discussion is still that "we need to pay more attention to visual evidence". The authors are recommended to provide deeper exploration on the implications of visual ignorance. For example, what algorithms or what special solutions can we design if we know the visual ignorance issue?
A4. We thank the reviewer for this suggestion. The Discussion already distinguishes whether visual information is available from whether it actually affects the output, and it gives concrete suggestions for training and evaluation. We agree that the paper does not yet provide a concrete solution. Below we provide a possible direction and discuss its implications.
A useful way to frame visual ignorance is through shortcut learning [1]: the tendency of deep networks to learn decision rules that work on the training distribution but fail when the test distribution requires information the shortcut bypasses. Because shortcuts can survive standard in-distribution evaluation, high benchmark accuracy alone may not reveal their use. Visual ignorance has exactly this structure. The model can answer from question-answer priors or category-level expectations, which are sufficient for much of the training data, and the decisive visual detail is never forced to influence the output. Our progressive blur and text-only evaluations reduce or remove visual evidence while retaining possible shortcuts based on the question and learned language priors. The high invariance rates are consistent with continued reliance on these shortcuts even when visual evidence is reduced or absent. This is consistent with VQA models relying on question–answer priors [2], contrastive vision–language models doing well on retrieval without binding objects to attributes and relations [3], and memorized world knowledge competing with conflicting visual input in modern VLMs [4]. We treat this as a hypothesis about where the bias comes from, not as a proven general rule for every model.
This framing makes a testable prediction: if the shortcut is the problem, then supervision that makes the shortcut insufficient should reduce visual ignorance. We tested this hypothesis with the VisOnlyQA dataset [5]. Its labels depend on precise visual attributes and relations: small changes in geometry or spatial layout can change the answer, so category-level priors are not enough. We fine-tuned the full Qwen3.5-9B model for 300 optimization steps with an effective batch size of 18, then repeated the causal experiment on the same VLMBias normal-counterfactual pairs. The intervention effect grows in the mid-to-late layers:
| Mean causal effect | Base | VisOnlyQA |
|---|---|---|
| Layers 16–23 | 0.304 | 0.429 |
| Layers 24–31 | 0.714 | 0.832 |
The stronger mid-to-late effects come with a substantial gain on VLMBias and a non-trivial gain on HallusionBench:
| Benchmark | Base | VisOnlyQA |
|---|---|---|
| VLMBias accuracy | 14.83% | 24.93% |
| HallusionBench question accuracy | 47.91% | 51.21% |
These results support training with supervision that depends on precise visual attributes. The model responds more strongly to inserted counterfactual signals in the mid-to-late layers and improves on both VLMBias and HallusionBench. Together, these results demonstrate that training on more fine-grained supervision signals may help alleviate the visual ignorance problem.
At inference time, the same logic gives a simple diagnostic signal. If a model's answer stays invariant after a controlled corruption of the image, the prediction may rely on memorized priors rather than the detail the question asks about. Practitioners can use this signal to flag unreliable predictions before acting on them.
More broadly, visual ignorance is not a single-dataset problem. It is a research problem that spans data, objectives, and evaluation: training signals that depend on the key visual detail, objectives that reward sensitivity to that detail, and metrics that distinguish real use of evidence from coincidental correctness. We see these results as one step in that direction; much work remains across tasks, architectures, and model scales.
[1] Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence 2020.
[2] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.
[3] When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It? ICLR 2023.
[4] Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts. EMNLP 2025.
[5] VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information. COLM 2025.
I thank the authors for the detailed rebuttal. However, some of my concerns remain.
W1. The evaluation method seems not very relevant to the previous context (as also noted by Reviewer YPAV).
Rebuttal: "The two analyses address the same main problem: whether answer-relevant visual evidence is effectively used in answering."
What I expected is that: the paper does some analysis first and then propose a new evaluation based on the analysisi. Here, the second part (the evaluation) is not very dependent on the first part (the analysis) and they look like two different analyses. But I think I can accepet this, and in this case, the paper just proposes two things for the same problem.
W2 & 3. The evaluation in lens 2 is not very novel and after a mild Gaussian blur, I think some images are still informative to answer the questions (as also noted by Reviewer 7qMu).
Rebuttal: The authors mainly focus on the portion of the models' answers that remain unchanged and in the appendix, present other evaluation techniques, including image removal.
My concern is that the Gaussian blur evaluation doesn't look better than previous methods. For visual ignorance, can we just use image removal or use some blank images to test LVLMs? It seems that the Gaussian blur is indeed a different method but not better than these simple ways.
W4. I think there might be some misunderstanding in my comment. I was not saying that the paper doesn't give concrete suggestions for training and evaluation. The analyses that the paper proposes are interesting but they do not provide something new, something different from previous works. The suggestions are not specific to visual ignorance, but general to LVLMs.
As I comment in the review, "Previous works already show that VLMs and benchmarks need to pay more attention to visual evidence. Visual ignorance is interesting because it is a special case where a model actually capture the task-relevant visual evidence. It tells us that capturing and utilizing visual evidence are actually two different things, but unfortunately it doesn't give us more insights."
I appreciate the authors for the new experiment and the detailed response. I think the topic is overall interesting and the authors could rethink about the lens 2 and go deeper into it.
Authors' Rebuttal (1/2)
We thank the reviewer for their valuable feedback.
W1. First, this evaluation method seems not very relevant to the previous context. The authors study on visual ignorance at the beginning, where a model can represent task-relevant visual evidence but do not use it to determine the answer. However, in the Gaussian-blur evaluation, the images are blurred and the model may neither represent visual evidence nor use it for answering. Thus, I'm not sure how the Guassian blur method is related to the visual ignorance. It looks like a general method to test the VLMs' robustness to visual inputs.
A1. We apologize for not making the connection clear. An unchanged answer under mild blur can reflect robustness when sufficient answer-relevant information remains. Our evaluation tracks predictions through the entire degradation sequence and measures how much benchmark credit is retained by answers that remain unchanged as visual detail is progressively removed. We agree that this behavioral analysis does not independently identify encoded-but-unused information in each example.
The two analyses address the same main problem: whether answer-relevant visual evidence is effectively used in answering. The internal analysis shows that decodable answer-relevant information need not produce the appropriate answer update. The benchmark analysis examines the evaluation consequence: substantial benchmark credit can persist even when predictions remain unchanged throughout progressive visual degradation. Appendix J further reports that more than 40% of answers remain unchanged after image removal for each of the three models, showing that many predictions persist even without image input. Together, these analyses connect a limitation in the use of represented information to the ability of benchmarks to reward visually insensitive predictions. We will clarify this connection in the revised manuscript.
W2. Second, the evaluation in lens 2 is not very novel. Prior works discuss a few ways to test VLMs, such as replacing, editing, removing, and deteriorating images or videos in benchmarks.
A2. We agree that image corruption alone is not novel when evaluating VLMs. However, as discussed in Section 2, we mainly focus on the portion of the models' answers that remain unchanged throughout multiple blur levels. This design has two benefits: (1) Progressive Gaussian blur provides a sequence of increasingly degraded inputs, allowing us to track whether predictions respond as visual detail is removed; (2) Consecutive agreement requires the prediction to match the original answer at every degradation level, excluding cases where it changes and later returns to that answer. Our contribution is a progressive evaluation protocol that measures both the prevalence of predictions that remain unchanged throughout visual degradation and the benchmark credit these predictions retain across twelve benchmarks.
W3. Last, after a mild Gaussian blur, I think some images are still informative to answer the questions. Thus, this evaluation seems to actually test the VLM robustness to visual evidence instead of ignorance. Why not just remove the images or replace them with all-zero tensors?
A3. We apologize for not explicitly mentioning other evaluation techniques in the main text. In Appendix J, specifically in Appendix Table 2, we present other evaluation techniques, including image removal. We observe that the averaged proportion of invariant answers before/after image removal is over 40% for the 3 models, which supports the results of progressive blur. Image removal tests whether predictions persist in the complete absence of image input. Progressive blur additionally tracks whether an answer remains unchanged at every stage of visual degradation. Agreement between the original-image and text-only answers alone cannot distinguish sustained invariance from a prediction that changes during degradation and later returns to the original answer. We therefore use image removal as a complementary control for the progressive evaluation. In the revised manuscript, we will mention the text-only evaluation explicitly in the main text.