acceptodds
Under review as a conference paper at ICLR 2027

Mirage Probes: How Vision Models Fake Visual Understanding

Abstract

Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using *Mirage Probes*, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from image-present internal activations across four target sites in two open-source VLMs. A Naive Bayes text baseline fails to recover this signal, ruling out surface lexical confounds. Analysis of per-question text-only answerability and further latent space probing expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model’s visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.

Primary area:
interpretability and explainable AI
Keywords:
VLMs, VQA, Mirage, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JqnaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the Mirage phenomenon in vision-language models (VLMs), where models can generate confident responses to visual questions even when they cannot access the corresponding images. The authors propose Mirage Probes, a contrastive probing framework that aims to study how this behavior is represented in the internal activations of VLMs.

This framework generates rephrased versions of the same question and identifies Mirage-like and Non-mirage-like responses to construct contrastive samples. Subsequently, the authors extract activations from residual streams, MLP outputs, post-attention outputs, and individual attention head outputs, and train different types of probes.

Experiments on multiple VLMs and visual question answering benchmarks show that the models' internal activations can be used to predict Mirage-related behaviors, and probes based on activation differences achieve higher accuracy. The authors further propose that Mirage may involve two distinct regimes, namely Textual Biases and Spurious Images, and study these phenomena through additional probing and intervention experiments.

Critical strengths
  1. The research motivation is meaningful. This study examines an important issue in VLMs: whether the models' seemingly correct visual responses are truly based on visual evidence. Studying the Mirage phenomenon from the perspective of the internal representations of models is a worthwhile direction that can help us further understand the limitations of current VLMs.

  2. The experimental design is reasonably sound and the evaluation scope is relatively broad. The authors attempt to reduce the differences in content between positive and negative samples by constructing contrastive samples, which is a valuable design. In addition, the authors study various activation positions and probing strategies, and conduct evaluations on multiple models and benchmarks. The supplementary intervention experiments further indicate that some of the identified internal representations are indeed related to Mirage-related behaviors.

Critical weaknesses
  1. Prompt changes may have an impact on Mirage behavior. In Section 3.2, the authors construct contrastive samples by generating four rewritten versions of each original question. However, the same image and semantically similar questions can produce two different types of responses, Mirage-like and Non-mirage-like, which raises questions about the influence of prompt changes. It is not yet clear whether these differences reflect different internal representations related to Mirage or whether the model is merely sensitive to different question formulations. In particular, even if the meaning of the question remains unchanged, certain rewrites may alter the model's confidence in its answers or its tendency to refuse to answer. Therefore, I believe it is necessary for the paper to further analyze how prompt changes affect the Mirage labels and whether the constructed contrastive samples can reliably reflect the behavior that the authors intend to study.

  2. The authors do not adequately explain why model-generated answers need to be included during activation extraction. As shown in Figure 3, the authors feed the complete conversation, including the model-generated answers, back into the VLM to extract internal activations. However, the paper does not adequately explain the motivation for doing so. Since the Mirage labels are also determined by comparing model answers, the extracted activations may contain information related to the answers, such as answer content, confidence, or refusal patterns. Therefore, higher probing accuracy does not necessarily mean that the probe has identified the true internal mechanism that causes Mirage. I think the authors should add experiments that use only activations extracted before answer generation to further clarify this issue.

  3. The generalization ability of the trained probes is insufficiently evaluated. The paper mainly uses data from different splits of the same benchmark to train and evaluate the probes. Although the authors use held-out test samples and extend the experiments to more models and datasets, such an evaluation may not be sufficient to prove that the probes have learned general Mirage-related representations rather than patterns specific to certain benchmarks. In particular, it is not clear whether a probe trained on one benchmark can remain effective on an unseen benchmark. I suggest that the authors add cross-benchmark evaluations, for example, by training a probe on one dataset and then directly testing it on other datasets without retraining. Such experiments can provide stronger evidence for the generalization ability of the proposed framework.

Questions for authors

Q1. Influence of prompt variations. In Section 3.2, the authors construct contrastive samples by rewriting the same question in different ways. Have the authors analyzed whether the observed Mirage/Non-Mirage differences are mainly caused by prompt sensitivity? For instance, could certain rewrites potentially alter the model's confidence or tendency to refuse to answer, rather than truly changing the model's reliance on visual information? An additional analysis of the impact of different prompt rewrites on the final labels would help clarify this issue.

Q2. Motivation for response-conditioned activation extraction. In Figure 3, why do the authors choose to include the model's generated answer during activation extraction instead of only using the image and question? Have the authors tried using only activations extracted before answer generation for probing and comparing the results of the two settings? Such a comparison would help determine whether the probe captures Mirage-related internal representations or mainly reflects the information already contained in the generated answer.

Q3. Cross-benchmark generalization. Can the authors further explain whether the trained probes can generalize across different benchmarks? For example, can a probe trained on VQA-RAD be directly used on MMMU-Pro or MedXpertQA without retraining? Although the paper includes evaluations on multiple benchmarks, conducting cross-benchmark transfer tests on the main activation probes would provide stronger evidence that the identified representations are not specific to a particular dataset. Additionally, could the authors clarify whether all the rewrites of the same original question are strictly kept within the same data split?

Reviewer_A13EOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that the "mirage" phenomenon, in which VLMs confidently answer image-based questions even without an image, is in fact a mixture of two mechanisms: textual bias, where the model answers from textual priors, and spurious images, where the model constructs fake visual information in latent space. The authors derive mirage labels by comparing responses with and without the image, reduce confounds using paraphrase-based contrastive pairs, and then train probes on various activations of the model. They claim that the trained classifiers can linearly distinguish mirage behavior well.

Critical strengths
  • The authors systematically construct a dataset of contrastive pairs to analyze the mirage phenomenon. Using this dataset and two models, they explore various internal representations and probing strategies, and show that mirage behavior can be linearly distinguished.
Critical weaknesses
  • Overall, the paper reads like a well-written experimental report. The mirage phenomenon raised as a problem of VLMs has already been identified in prior work, and the approach adopted to analyze it, training linear classifiers on internal representations, has also been proposed in other papers (e.g., the works cited in the related work section & https://arxiv.org/abs/2509.03025).
    • Therefore, the paper's contribution should lie in what it means that the probes separate the classes well, and how this can be used to improve the model. However, all related content appears only in the appendix, and its scale is small.
    • There is no actual mitigation method or experiment showing performance improvement, and the main text stops at confirming decodability. Apart from this, it is difficult to see substantial novelty in the paper itself.
  • The authors claim that the mirage phenomenon is a mixture of two mechanisms, but most of the main text focuses on whether mirage behavior can be detected and on the related experimental settings. Evidence for the two mechanisms is not presented in sufficient detail.
    • The evidence in the main text is weak relative to how strongly the claim is emphasized in the abstract.
    • Similarly, the evidence for the core claim that "mirage behavior is linearly decodable" is weak in some settings.
      • The accuracy of contrastive logistic regression, the main supporting evidence, is only around 70%, and in some cases it is close to chance.
Questions for authors

Please refer to the weaknesses section. It would be nice to provide additional experiments related to the second point.

Reviewer_aYmJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why vision-language models sometimes answer visual questions confidently even when the image is missing, a behavior referred to as a "mirage." The authors introduce Mirage Probes, which uses paraphrased question pairs and linear probes to study how this behavior is represented in VLM activations.

Critical strengths
  1. I think the paper asks an interesting question. A model giving the correct answer does not necessarily mean it actually used the image, and understanding this distinction is important for evaluating VLMs. Looking at internal representations rather than only the final answers is a useful direction.

  2. The experimental setup is fairly thorough. The authors examine multiple activation types, compare different probing strategies, and use paraphrased question pairs to reduce textual confounds. I also appreciate that the paper includes additional causal experiments and reports when they do not support the proposed interpretation.

Critical weaknesses
  1. My main concern is whether the probes are really identifying an internal mirage mechanism or simply learning differences already introduced by the labeling procedure. The mirage labels are defined by comparing model responses with and without images, and the strongest results come from difference probes that also use the activations from these two conditions. For example, Table 1 reports accuracies above 95% on VQA-RAD, while the image-present logistic regression probes are substantially weaker. Since the label itself depends on how the model responds when the image is removed, I am not sure how much these high scores tell us beyond detecting that same difference in latent space. The activation extraction also includes the generated responses, which may introduce additional response-level cues. The Naive Bayes baseline is helpful, but it does not seem fully comparable to a probe that has access to both image-present and image-absent activations. A stronger text-based baseline using both responses, or an experiment controlling for the generated response during activation extraction, would make this result much more convincing.

  2. I am also not fully convinced that the experiments establish two distinct mirage mechanisms. The distinction between textual biases and spurious images is interesting, but the classification largely relies on whether the generated response appears to fabricate visual information or answer from text. The authors then show that these categories can be separated using VLM activations. However, separability alone does not necessarily mean that the model constructs a false visual representation internally. The causal experiments in Appendix H provide some evidence that mirage-related directions affect model behavior, which is useful, but Appendix J reports no clear causal effect for the proposed two-mechanism distinction. I think the current results support two different behavioral patterns more clearly than two distinct underlying mechanisms. This distinction matters because the two-mechanism explanation is one of the paper's main contributions.

Questions for authors

Section 3.5 appears to select the best held-out test accuracy across attention heads and layers. Could the authors clarify whether this selection is performed on validation data? If the best result is selected directly from test performance, the reported probing accuracy could be optimistic.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.