acceptodds
Under review as a conference paper at ICLR 2027

SightLine: What a Vision-Language Model Reads From an Image Is Not What It Answers From

Abstract

Look inside a vision-language model and you can often find what it saw in the image. It is natural to conclude that the model answers from what we found there. We show that it does not: what we read out is the vision encoder's description of the image, and the language model answers from somewhere else. SightLine tests use by intervention rather than by reading: we install a different value in the model's state, verify with an independent probe that it is present, and check whether the answer follows. In a controlled legend-lookup task, the probe reads the label assigned to a symbol from the symbol's own visual tokens with perfect accuracy, and the vision encoder writes it there within two blocks. A different label installed at those tokens is verified present, yet the answer never follows it: the model overwrites the installed label by mid-stack and keeps its original answer on every pair, even when the label is re-imposed at every layer. The answer depends instead on the printed label in the legend, on eight attention heads at the answer position, and on a layout-specific row identity at the symbol's own tokens that tells the model which legend row it is looking at. That identity is a handle: injected on Qwen2.5-VL-7B with no training, it switches the answer to the predicted label on 68% and 88% of held-out pairs across the two legend layouts, and back. The separation and the attention-head pathway replicate causally in three model families. On real charts the answer still depends on the printed label, the row identity is readable at legend swatches, and errors arise after a correct identity. The same pattern appears on natural benchmarks: an image-quality feature is readable in four model families, survives changes of degradation and is already present at the vision-language connector, yet predicts nothing about which items the model gets wrong. Overwriting, not reading, is the test of use. More experiments and code are provided in the appendix and https://anonymous.4open.science/r/sightline-supplement/.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, vision-language models, activation patching, causal intervention, probing, representation analysis, visual grounding, binding, chart understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3sRvOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzed how vlm will read the visual tokens for vqa questions and revealed that vlm often looks at regions where we might not expect by ablations like replace embeddings from "donor" images or by introducing "slot change vector" to direct vlm readouts.

Critical strengths
  • An interesting surgery paper to reveal where vlm is looking inside when trying to answer vqa questions.
Critical weaknesses
  • Tokens after vision encoder have already exchanged information. The core question behind the scene is
    • 1\ how text tokens will attend to visual tokens
    • 2\ which information is stored in the visual tokens
    • The current study can be more thorough
      • how's the attention map looks like?
      • if we replace letters with symbols e.g. literally put donor's star to base letter, will model read the letter information stored in the donor symbol?
  • Limited novelty. What's additional contribution this paper is trying to provide? We switched the embeddings of a patch, then get the information out from that switched embedding.
    • the probing is misleading. it's a trained classifier to read that particular patch of embeddings to a target label, which only proves that the embedding is linear separable in its embedding space for the labels, but has nothing to do with which information in that space will be leveraged by the vlm readout.
Questions for authors

The paper looks over AI-polished e.g. "Look inside a vision–language model and you can often find what it saw in the image. It is natural to conclude that the model answers from what we found there. We show that it does not: what we read out is the vision encoder’s description of the image, and the language model answers from somewhere else. SightLine tests use by intervention rather than by reading.."

which makes the reading a bit tedious.

Reviewer_AeTfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether a model actually uses the information we can read from its internal states. The authors test that idea by putting a different label into the circle’s representation. They then use an independent probe to check that different label really is there before looking at the final answer. What they find is that the model still answers the original label. The inserted label gets overwritten later, and even putting it back at every layer barely changes the answer. They also find another signal that seems to tell the model which row to look at. Changing that signal can make the model read a different row and give a different answer, at least in Qwen2.5. So the model seems to use the symbol to find the right row, then read the letter from there. That is what makes the result interesting: the label is already sitting in the symbol’s representation, and we can read it perfectly, but the model seems to get its answer through a different route.

Critical strengths
  1. It is good to see how the paper takes a common assumption and tests it directly. It is easy to assume that if we can read information from a model’s internal states, the model must be using it. The paper gives a clear example of why that assumption can fail.

  2. The intervention setup is reasonable. The authors check that the new label is actually readable after they write it into the model. This makes the unchanged answer much more informative, since it helps rule out the possibility that the intervention simply failed.

  3. The authors put effort into checking alternative explanations. They repeat the intervention at every layer, test larger regions, and compare against interventions that do change the answer. The random and shuffled controls, along with the reverse row-identity intervention, also make the control results more convincing.

  4. The paper gives us useful positive evidence about how the answer is formed. It identifies the printed label, a small group of attention heads, and a row-identity signal as important pieces of the process. The key/value separation and readout findings also hold across three models.

Critical weaknesses
  1. The authors can read out the correct row identity in many cases where the model answers incorrectly, so they suggest that something goes wrong afterward. But their main argument is that being able to read information does not mean the model uses it. The same concern applies here: the correct row identity might be available without actually guiding the answer.

  2. The row-switching result is interesting, but it only works reliably in Qwen2.5. It does not produce the predicted answer changes in Qwen3, and the Gemma result does not beat the random control. This makes me wonder how much of the proposed mechanism carries over to other models.

  3. Most of the strongest evidence comes from a very controlled matching task with only a few legend entries. We still do not have the full causal story. Making the model change its answer by injecting a signal shows that the signal can influence it. It does not necessarily show that the model normally needs that signal. I would like to see a more direct test connecting row identity to the selected label and then to the final answer.

Questions for authors

My question is whether the probe only tells us what it can read. It confirms that the new label is there, but the model might use a different encoding of the same information or get it from somewhere else. The authors test several alternatives, which helps.

Reviewer_2aMSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SightLine, a method called certified installation for testing whether information encoded in a VLM is actually used to produce its answers. Rather than relying on probing alone, the method intervenes on internal representations, verifies that the intended information has been installed, and checks whether the model's output changes. Experiments on synthetic legend-lookup tasks show that labels can be decoded from a symbol's visual tokens without those labels being causally used to determine the answer. Instead, the model relies on the printed legend label and a separate row-identity signal. The findings are tested across three model families and extended to real charts and natural-image benchmarks.

Critical strengths
  1. Certified installation provides a compelling alternative to conventional probing by explicitly verifying that an intervention succeeds before drawing conclusions about causal use.
  2. Pixel-identical counterfactual pairs, held-out probes, matched controls, and bidirectional interventions help isolate the mechanisms underlying model predictions.
  3. The paper traces information through the vision encoder, language model layers, attention heads, and answer-generation position, offering a clear account of the observed behavior.
  4. Several key findings are replicated across three model families, strengthening the evidence that the observations are not specific to a single architecture.
  5. The real-chart experiments suggest that some errors arise from incorrect legend-to-symbol binding downstream, even when the relevant row identity is correctly represented.
Critical weaknesses
  1. Although row-identity injection changes answers substantially in Qwen2.5-VL, the corresponding behavioral control does not replicate on Qwen3-VL. The evidence for this mechanism is therefore less general than the paper's core findings.
  2. The experiments identify important components of the answer pathway, but do not fully establish how the row identity is computed or whether it is necessary for correct predictions.
  3. The central conclusions depend on the chosen activation-patching and representation-estimation methods. Although the authors investigate alternative interventions, the results do not completely rule out other representations or pathways that could explain the observed behavior.
  4. The controlled experiments focus heavily on legend lookup, and the natural-benchmark results provide less direct causal evidence. It remains unclear how broadly the proposed mechanism generalizes to other forms of visual reasoning.
Questions for authors

Please see the mentioned weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.