- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WHATIFVIS to study whether MLLMs can controllably use or ignore visual evidence that conflicts with prior knowledge. The authors combine behavioral evaluation with reconstruction, probing, fine-tuning, activation patching, and rank-1 steering. The results suggest that coarse visual information can remain available in model representations even when models fail to use it appropriately.
Critical strengths
The paper studies an important failure mode with a well-designed paired benchmark and several complementary analyses. I especially like that the authors go beyond behavioral accuracy and test whether conflicting visual information remains recoverable in the representations.
Critical weaknesses
Recoverability through a trained reconstruction model does not necessarily imply that the original MLLM can readily access or use the same information during inference. Also, the benchmark treats annotated world-knowledge answers as each model’s parametric prior, which may conflate context control with differences in factual knowledge.
Questions for authors
Have the authors considered filtering examples based on whether each model can correctly answer the prior-knowledge question without the image? This would help separate failures of context control from failures of parametric knowledge.
3 comments
Response to Reviewer ebdd (2/2)
Q1: Filtering examples by prior knowledge.
We ran the proposed filter. For each model we keep a pair only if the model ranks the annotated prior above the counterfactual answer without the image, and recompute paired accuracy on the kept pairs.
Paired accuracy on all pairs → pairs whose prior the model knows (%, macro average over the five test sets):
| Model | Vanilla PairProb | SFT PairProb |
|---|---|---|
| Qwen2.5-VL-3B | 5.4 → 5.5 | 52.6 → 54.9 |
| Qwen2.5-VL-7B | 8.4 → 8.4 | 57.8 → 59.7 |
| Qwen3.5-4B | 18.2 → 18.4 | 64.9 → 66.7 |
| Qwen3.5-9B | 13.9 → 14.4 | 65.2 → 68.8 |
| Gemma-4-E2B-IT | 15.6 → 16.4 | 52.8 → 54.5 |
| Gemma-4-E4B-IT | 23.1 → 23.9 | 59.4 → 63.4 |
| Mean | 14.1 → 14.5 | 58.8 → 61.3 |
Filtering changes paired accuracy by at most 4 points for any model, before or after SFT, because few pairs are removed. Failures of parametric knowledge therefore account for very little of the gap.
Thank you again for your constructive feedback. We are happy to answer any further questions.
Thank you for your new strong results. I will maintain my positive score.
Response to Reviewer ebdd (1/2)
We thank you for the positive comments, thoughtful feedback, and insightful questions. We respond to each point below.
We agree, and the paper draws the same distinction: Section 4 states that reconstruction “establishes recoverability of counterfactual evidence, without establishing that the model’s computation uses it” (Line 271). We use reconstruction only to argue against complete information loss (H1). Evidence that is retained but not readily used by the model falls under H2 in our framing.
There are two results that matter more directly for accessibility:
(i) Linear probing at the answer position (Section 4, Figure 5). A linear classifier is a much weaker readout than a trained decoder. Applied to the final text-prompt position, where the answer is produced, it predicts the counterfactual information with 59.6–72.6% accuracy across three models (chance 25%; shuffled-image and no-image controls stay near chance). Part of the information therefore reaches the answer position in linearly decodable form, although the models’ direct answers on the counterfactual images are near zero, consistent with [1]. (ii) Interventions on the language backbone. With the vision encoder frozen, SFT raises mean paired accuracy from 14.1% to 58.75%. A rank-one direction applied to the vanilla models, with no change to their weights, raises it to 34.2%. The vanilla models can thus act on the visual evidence under a small intervention. Neither result shows that all of the reconstructed information is accessible, which is why we claim only that the evidence argues against complete information loss. This is the logic of the paper: since the evidence reaches the answer position but is not used as instructed, we tested whether the language backbone can be adapted (SFT) or steered (a rank-one direction) to use it.
Thank you for this suggestion. We added a closed-book check (Table 8, Line 1890): each vanilla model answers the question with no image, statement or instruction, and we test whether it ranks the annotated prior above the counterfactual answer. This is the property the benchmark relies on, namely that the model’s own prior favors the annotated answer over the counterfactual one. Closed-book accuracy (%) on the test splits, as probability:
Under the probability scorer, the models favor the annotated prior on 96.2% of items on average, and on at least 85% for every model and task. Exact match is lower (avg 66.7%) because it penalizes differences in wording and format that the probability scorer ignores (Line 1277).
The same table shows that the failures come from context control. On Color, the models hold the annotated prior on 96.4% of items without the image, but give it on only 36.8% when the image is present and they are told to ignore it (Size: 92.6% versus 32.0%). The two hardest tasks, Count and Weight, are also ones where the prior is held almost perfectly (95.4% and 99.1%). The low paired accuracy in Tables 5 and 7 therefore reflects how the models handle the context, not missing parametric knowledge.
This matches the design of the benchmark, which uses coarse, common-knowledge facts so that the annotated prior is unlikely to differ from the model’s. Prior text-only work on controllable context sensitivity likewise takes the dataset’s answers as the prior [2].
[1] Vo, An, et al. “Vision language models are biased.” arXiv preprint arXiv:2505.23941 (2025).
[2] Minder, Julian, et al. “Controllable Context Sensitivity and the Knob Behind It.” ICLR 2025.