acceptodds
Under review as a conference paper at ICLR 2027

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

Abstract

How reliably can we expect a multimodal large language model (MLLM) to use or disregard an image that conflicts with prior knowledge? We introduce the WhatIfVis benchmark, adapting and extending counterfactual datasets across five coarse-grained conflict types, such as color and size. Each example holds the image and question fixed while changing only the instruction to use or ignore the image. Across six MLLMs, mean paired accuracy is only : success requires ranking the intended answer above the alternative under both instructions. Yet this failure is ambiguous: visual evidence may be missing from the model's representations (H1), or remain available but be mishandled (H2). Reconstruction models trained without counterfactual examples recover coarse attributes from the final-layer image tokens of three frozen MLLMs, arguing against H1 for these attributes. LoRA adaptation of the language backbone on one conflict type, with the vision encoder frozen, raises mean paired accuracy to , with gains extending to four held-out types. Activation patching identifies layer windows where interventions change whether answers follow the image or prior knowledge. A learned rank-1 intervention transfers from each adapted model to its original, adapter-free counterpart, reaching mean paired accuracy without verbal intent instructions. Matched text statements make the same task-relevant facts easier to follow when instructed, providing a comparison that also reflects differences in evidence extraction. MLLMs can retain visual evidence yet fail to control its use. WhatIfVis exposes this gap and provides a testbed for developing more robust multimodal models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision language model, context and prior knowledge

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ebddOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WHATIFVIS to study whether MLLMs can controllably use or ignore visual evidence that conflicts with prior knowledge. The authors combine behavioral evaluation with reconstruction, probing, fine-tuning, activation patching, and rank-1 steering. The results suggest that coarse visual information can remain available in model representations even when models fail to use it appropriately.

Critical strengths

The paper studies an important failure mode with a well-designed paired benchmark and several complementary analyses. I especially like that the authors go beyond behavioral accuracy and test whether conflicting visual information remains recoverable in the representations.

Critical weaknesses

Recoverability through a trained reconstruction model does not necessarily imply that the original MLLM can readily access or use the same information during inference. Also, the benchmark treats annotated world-knowledge answers as each model’s parametric prior, which may conflate context control with differences in factual knowledge.

Questions for authors

Have the authors considered filtering examples based on whether each model can correctly answer the prior-knowledge question without the image? This would help separate failures of context control from failures of parametric knowledge.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ebdd (1/2)

We thank you for the positive comments, thoughtful feedback, and insightful questions. We respond to each point below.

W1: Recoverability versus accessibility.

We agree, and the paper draws the same distinction: Section 4 states that reconstruction “establishes recoverability of counterfactual evidence, without establishing that the model’s computation uses it” (Line 271). We use reconstruction only to argue against complete information loss (H1). Evidence that is retained but not readily used by the model falls under H2 in our framing.

There are two results that matter more directly for accessibility:

(i) Linear probing at the answer position (Section 4, Figure 5). A linear classifier is a much weaker readout than a trained decoder. Applied to the final text-prompt position, where the answer is produced, it predicts the counterfactual information with 59.6–72.6% accuracy across three models (chance 25%; shuffled-image and no-image controls stay near chance). Part of the information therefore reaches the answer position in linearly decodable form, although the models’ direct answers on the counterfactual images are near zero, consistent with [1]. (ii) Interventions on the language backbone. With the vision encoder frozen, SFT raises mean paired accuracy from 14.1% to 58.75%. A rank-one direction applied to the vanilla models, with no change to their weights, raises it to 34.2%. The vanilla models can thus act on the visual evidence under a small intervention. Neither result shows that all of the reconstructed information is accessible, which is why we claim only that the evidence argues against complete information loss. This is the logic of the paper: since the evidence reaches the answer position but is not used as instructed, we tested whether the language backbone can be adapted (SFT) or steered (a rank-one direction) to use it.

W2: Annotated answers versus each model’s parametric prior.

Thank you for this suggestion. We added a closed-book check (Table 8, Line 1890): each vanilla model answers the question with no image, statement or instruction, and we test whether it ranks the annotated prior above the counterfactual answer. This is the property the benchmark relies on, namely that the model’s own prior favors the annotated answer over the counterfactual one. Closed-book accuracy (%) on the test splits, as probability:

ModelColorCountSizeS&TWeightAvg.
Qwen2.5-VL-3B96.792.585.097.396.493.6
Qwen2.5-VL-7B96.792.594.096.399.695.8
Qwen3.5-4B100.096.695.397.7100.097.9
Qwen3.5-9B100.097.997.397.7100.098.6
Gemma-4-E2B-IT89.896.687.798.398.494.1
Gemma-4-E4B-IT95.396.696.398.7100.097.4
Mean96.495.492.697.799.196.2

Under the probability scorer, the models favor the annotated prior on 96.2% of items on average, and on at least 85% for every model and task. Exact match is lower (avg 66.7%) because it penalizes differences in wording and format that the probability scorer ignores (Line 1277).

The same table shows that the failures come from context control. On Color, the models hold the annotated prior on 96.4% of items without the image, but give it on only 36.8% when the image is present and they are told to ignore it (Size: 92.6% versus 32.0%). The two hardest tasks, Count and Weight, are also ones where the prior is held almost perfectly (95.4% and 99.1%). The low paired accuracy in Tables 5 and 7 therefore reflects how the models handle the context, not missing parametric knowledge.

This matches the design of the benchmark, which uses coarse, common-knowledge facts so that the annotated prior is unlikely to differ from the model’s. Prior text-only work on controllable context sensitivity likewise takes the dataset’s answers as the prior [2].

[1] Vo, An, et al. “Vision language models are biased.” arXiv preprint arXiv:2505.23941 (2025).

[2] Minder, Julian, et al. “Controllable Context Sensitivity and the Knob Behind It.” ICLR 2025.

AuthorsOpenReview ↗

Response to Reviewer ebdd (2/2)

Q1: Filtering examples by prior knowledge.

We ran the proposed filter. For each model we keep a pair only if the model ranks the annotated prior above the counterfactual answer without the image, and recompute paired accuracy on the kept pairs.

Paired accuracy on all pairs → pairs whose prior the model knows (%, macro average over the five test sets):

ModelVanilla PairProbSFT PairProb
Qwen2.5-VL-3B5.4 → 5.552.6 → 54.9
Qwen2.5-VL-7B8.4 → 8.457.8 → 59.7
Qwen3.5-4B18.2 → 18.464.9 → 66.7
Qwen3.5-9B13.9 → 14.465.2 → 68.8
Gemma-4-E2B-IT15.6 → 16.452.8 → 54.5
Gemma-4-E4B-IT23.1 → 23.959.4 → 63.4
Mean14.1 → 14.558.8 → 61.3

Filtering changes paired accuracy by at most 4 points for any model, before or after SFT, because few pairs are removed. Failures of parametric knowledge therefore account for very little of the gap.

Thank you again for your constructive feedback. We are happy to answer any further questions.

Thank you for your new strong results. I will maintain my positive score.

Reviewer_5ri6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether MLLMs can follow instructions to use or ignore images that conflict with prior knowledge. It introduces WhatIfVis, a benchmark covering five conflict types, with paired evaluation requiring correct behavior under both instructions. Reconstruction and linear probing show that some counterfactual attributes remain recoverable even when models answer incorrectly. LoRA fine-tuning increases mean probability-based paired accuracy from 14.1% to 58.75%. A learned rank-one intervention transfers to the original models and reaches 34.2%.

Critical strengths

1.The paper discusses an interesting point: MLLMs should both follow conflicting visual evidence and disregard it when instructed. The paired evaluation is useful because success requires both behaviors on the same example.

2.Combining reconstruction, linear probing, and activation interventions provides useful evidence for separating information retention from information use. The results offer a concrete view of visual content preserved in hidden states.

3.The experiments cover six models, and training on one conflict type improves performance on four held-out types. Transferring a single learned direction from the adapted model to its original counterpart is also an interesting result.

Critical weaknesses

1.The image reconstruction/information retention results come from instruct-tuned MLLMs, which may have learned to encode similar visual patterns during alignment. Comparisons with earlier checkpoints (e.g., Qwen3.5-base) would help clarify whether the gap between information retention and answering is attributable to the alignment stage or training distribution.

2.The mechanism behind the LoRA gains is not directly examined. Patching attention outputs within SFT models does not establish whether fine-tuning changes attention to image tokens. A before-and-after analysis, together with a simple image-attention scaling baseline, would help test this explanation

3.The SFT gains may reflect both better instruction following and better visual understanding. Freezing the vision encoder does not separate these effects, since the language backbone can still learn to extract visual information or interpret visual relations more accurately. A matched SFT baseline trained only to answer from images, together with fixed-instruction visual evaluations, would make the contribution of instruction-dependent control clearer.

Questions for authors

Listed in the "Weaknesses" section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5ri6 (1/3)

Thank you for the positive assessment and the insightful suggestions. We respond to each point below.

W1: Comparing base and instruction-tuned checkpoints.

We agree this comparison would show at which training stage the gap between retention and answering arises, and whether the alignment creates or strengthens the retention. That is a question about how the gap develops over training, and it is complementary to our study. Our claims concern the released models as they are deployed. The counterfactual evidence is recoverable from their representations, but they do not use or ignore it as directed. These claims hold whichever training stage the gap emerges at (base, SFT, instruction etc). A stage-wise study would also need a different protocol for the answering side, since base checkpoints are not trained to follow instructions that the paired evaluation needs. We now state in the Limitations (Line 936) that we do not attribute the gap to a particular training stage, and name the comparison you suggest as future work.

AuthorsOpenReview ↗

Response to Reviewer 5ri6 (2/3)

W2: The mechanism behind the LoRA gains

We would like to clarify that SFT is used to test whether controllability is learnable and to locate intervention layers for steering (Section 5.2), and the paper’s conclusions do not rely on a particular mechanism for the LoRA gains. We nevertheless found the two analyses you propose a valuable addition, and we have included both as a supplementary analysis for the two Qwen2.5-VL models.

In short, SFT does change attention to image tokens, mainly by making it depend on the instruction. Suppressing attention to the image is sufficient to reproduce the ignore-image behavior in the vanilla model, but raising it does not reach the use-image accuracy of the SFT models. Attention to the image is therefore one identified component of the SFT gain, not a full account of it.

(i) Before-and-after attention analysis (Section I.2, Line 1939; Figure 31; Table 10). For each test prompt, we measure at every layer the share of the final prompt token's attention that falls on image tokens (mean over heads), before and after SFT, under both instructions, on all five test sets. For this analysis we retrained the SFT models of Qwen2.5-VL-3B and 7B. Figure 31 shows all layers. The table below reports the layer where attention under the use-image instruction peaks, which is the same layer before and after SFT.

Model (layer)Use-image (%)Ignore-image (%)Gap (use − ignore)
Qwen2.5-VL-3B (33 of 36), vanilla33.0329.49+3.54
Qwen2.5-VL-3B (33 of 36), SFT38.8411.10+27.74
Qwen2.5-VL-7B (27 of 28), vanilla24.7921.55+3.24
Qwen2.5-VL-7B (27 of 28), SFT33.3714.55+18.82

The vanilla models attend to the image almost equally under both instructions. After SFT, attention under the ignore-image instruction falls sharply in later layers.

(ii) Image-attention scaling baseline (Section I.3, Line 2021; Eq. 8; Table 11). In the vanilla model, we add to the pre-softmax attention scores from text queries to image keys, in every layer and head. This multiplies the odds of attending to the image by and leaves the distribution within the image and within the text unchanged; γ→0\gamma \to 0 removes the image. As with DAS, the model receives no intent instruction: we apply ctx when the intended behavior is to use the image and prior when it is to ignore it. Both constants are tuned on validation items of the five tasks, whereas DAS and SFT use Spatial-Temporal only, so the comparison below favors scaling.

Pair accuracy on the test sets, macro average (%, probability / exact match). Baseline, DAS, and SFT are from Table 5. Baseline and SFT use the intent instruction; DAS and scaling do not.

ModelBaselineDASScalingSFT
Qwen2.5-VL-3B5.42 / 2.5538.69 / 15.2044.98 / 26.1852.56 / 23.77
Qwen2.5-VL-7B8.40 / 3.3133.70 / 14.2047.71 / 20.0157.75 / 18.78

Scaling is above DAS on both models, below SFT under the probability scorer, and slightly above SFT under exact match. The accuracy under each intent (Table 11b) shows where this comes from.

Ignore-image. The selected prior (3.410-4 and 1.410-5) removes the image almost entirely, and accuracy then matches closed-book accuracy (93.73% vs. 93.59%; 96.48% vs. 95.83%). Suppressing attention to the image is thus sufficient to recover the prior answer, which is consistent with the suppression that SFT learns in (1).

Use-image. At the selected γctx\gamma_{\text{ctx}} (1.28 and 44.7), use-image accuracy is 50.29% and 51.10%. The instructed vanilla models reach 50.32% and 41.53%, so scaling gives no gain on 3B and +9.6 pp on 7B, and both stay well below the retrained SFT models (65.33% and 67.84%). For the 7B model this means that multiplying the odds of attending to the image by about 45 at every layer still does not reach the SFT accuracy. This is consistent with W3, where the use-image gain is reproduced by training to answer from images alone.

It’s also worth noting that Scaling and DAS are not equivalent interventions: scaling acts on every layer and head and, on the ignore side, amounts to removing the image, whereas DAS changes one direction at one layer and position with the image still available. We treat scaling as a measure of how far attention alone can go. Our analyses show that SFT suppresses attention to the image and that such suppression is sufficient in the vanilla model. They do not show that it is necessary for the SFT gain; testing that would require restoring image attention in the SFT models under the ignore-image instruction, which we leave to future work.

AuthorsOpenReview ↗

Response to Reviewer 5ri6 (3/3)

W3: Instruction following versus visual understanding.

We agree that freezing the vision encoder does not separate the two effects. We froze it so that the visual tokens entering the language backbone stay identical to those of the vanilla models whose retention we test in Section 4. To separate them, we trained the matched baseline you describe and evaluated all models under each fixed instruction (Section I.1, Line 1922; Table 9).

Image-answer-only SFT uses the same Spatial-Temporal training images, LoRA configuration, seed and number of steps as paired SFT, but every training sample has the use-image instruction and the image answer. We evaluate each model separately under the use-image and the ignore-image instruction, and report paired accuracy, which requires both to be correct.

Six-model macro average over the five test sets (%, probability / exact match):

ModelUse-image instructionIgnore-image instructionPaired accuracy
Vanilla46.5 / 36.267.5 / 55.514.1 / 6.6
Paired SFT63.8 / 48.182.9 / 69.558.8 / 25.6
Image-answer-only SFT63.9 / 47.138.6 / 31.04.9 / 3.0

Under the use-image instruction, image-answer-only SFT matches paired SFT (63.9% vs. 63.8%). This part of the SFT gain is therefore better answering from images, not better instruction following. The two-candidate scorer cannot tell whether it reflects better visual reading or a general preference for the image answer.

Image-answer-only SFT follows the image whatever the instruction says: accuracy under the ignore-image instruction falls from 67.5% to 38.6%, and paired accuracy drops below vanilla (4.9% vs. 14.1%). Paired SFT instead improves under both instructions and raises paired accuracy to 58.8%. Controllability, which is what the paper measures and claims, therefore comes from the instruction-dependent training and cannot be obtained by better image answering alone.

Thank you again for your constructive feedback. We are happy to answer any further questions.

As the discussion period draws to a close, we would be grateful for the reviewer’s view on whether our response to the weaknesses addresses the concerns. We recognize reviewers are busy, and we appreciate the time already spent on our submission.

Reviewer_xugLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WhatIfVis, a counterfactual benchmark assessing whether VLMs follow instructions to use visual evidence or prior knowledge. It examines whether failures comes from missing visual information (H1) or mishandling retained information (H2). Reconstruction and probing indicate that the tested visual attributes remain recoverable, arguing against complete information loss. SFT substantially improves controllability and generalizes across tasks, while learned subspace steering provides more limited gains when transferred to the original models without explicit intent instructions.

Critical strengths
  • The paper presents a coherent experimental framework for examining the two hypotheses. Reconstruction and probing assess information availability, while behavioral evaluation and interventions investigate its use. The purpose of each experiment is clear, and the analyses follow a logical progression.
  • The findings provide useful evidence that MLLMs can retain visual information while failing to use it as instructed. Cross-task improvements from SFT and the transfer of learned steering directions to the original models offer actionable insights into improving visual controllability.
Critical weaknesses
  • The methodological novelty is limited, as the study primarily combines established techniques, including MetaQuery-based reconstruction, linear probing, LoRA, and DAS. Its contributions lie mainly in the benchmark and their integration, rather than a substantially new analysis or intervention method.
  • The causes of the remaining failures on 'Count' and 'Weight' are insufficiently disentangled. The explanation that these tasks require additional reasoning is plausible, but the experiments do not clearly separate difficulties in visual evidence extraction, reasoning, and information-source selection. Since SFT uses only spatial-temporal examples, limited cross-task transfer is another possible explanation that warrants targeted analysis.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer xugL

Thank you for the positive assessment and the thoughtful feedback. We respond to each point below.

W1: Novelty from combining existing techniques.

We agree that our contribution is primarily empirical: the WhatIfVis benchmark with paired image and text evidence, a framework that separates missing visual information (H1) from mishandled information (H2), and the findings this gives. We did however adapt the existing methods so that they can test these hypotheses, rather than applying them off the shelf.

(i) Reconstruction as a test of H1. We remove MetaQuery’s learnable query tokens and condition the decoder only on the final-layer image-token representations of the frozen MLLM (Line 209). The connector and decoder are trained on ImageNet images only, with no counterfactual image or attribute label. The reconstruction therefore shows what visual information is still present at that point, without the task labels that linear probing needs.

(ii) Subspace control for visual context. We extend the DAS-based procedure of Minder et al.[2] from textual to visual context. The same procedure, with no architecture-specific changes, finds a rank-one direction in three architecturally distinct families (Qwen2.5-VL, Qwen3.5, Gemma-4; procedure in Appendix D, per-model results in Table 5). The intervention site is not shared across models: we localize it per model, and the selected layers and steering scales differ substantially (Appendix E.3, Table 6). The learned direction transfers from each adapted model to its original, adapter-free counterpart and raises mean PairProb from 14.1% to 34.2% without intent instructions. To the best of our knowledge, this is the first use of DAS to learn a visual context-control direction in MLLMs.

W2: Causes of the remaining failures on Count and Weight.

We agree the “further reasoning” explanation was stated too strongly, and we now present it as a hypothesis (Section 5.1, Line 360). To test the limited-transfer explanation, we compare each held-out task before and after SFT on Spatial-Temporal (S&T), in the image and text settings (from Tables 5 and 7). The table gives six-model mean PairProb (%) (vanilla → SFT).

SettingColorSizeCountWeight
Image32.3 → 96.613.6 → 55.22.4 → 32.90.7 → 22.1
Text49.5 → 99.528.8 → 87.760.2 → 97.721.8 → 82.9

Count and Weight reach 97.7% and 82.9% when the same counterfactual is given as text, with the same questions and the same held-out status. The held out status of these tasks during the training is therefore not sufficient on its own to explain the low image-setting results. The remaining gap is tied to the evidence being visual: extracting it, reasoning over it, or weighting it against the prior. As in Section 6, we treat text as a reference and not as a pure modality ablation, so this comparison does not separate these three, and it does not exclude limited transfer of visual skills from S&T images. We will say so in the paper (appendix H). For Count, reconstruction and probing (Section 4) show that the counterfactual count is recoverable from the representations, which argues against complete information loss. Fully separating extraction from reasoning needs task-specific study, as Pang et al. [1] show for counting alone; we will state this as a limitation and future work (Line 945).

Thank you again for your constructive feedback. We are happy to answer any further questions.

[1] Pang, Xingzhou, et al. "Unveiling the Visual Counting Bottleneck in Vision-Language Models." arXiv preprint arXiv:2605.30170 (2026).

[2]Minder, Julian, et al. “Controllable Context Sensitivity and the Knob Behind It.” ICLR 2025.

As the discussion period draws to a close, we would be grateful for the reviewer’s view on whether our response to the weaknesses addresses the concerns. We recognize reviewers are busy, and we appreciate the time already spent on our submission.

Thank you for the clarifications. While the methods are largely based on existing analysis tools, the authors resolved the remaining concerns and the main takeaway is interesting and helpful to further understand the internal mechanism of MLLMs. Thus I will keep my positive rating.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. If you haven't already, we would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.

Thank you for your time and valuable contributions to the review process!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.