SightLine: What a Vision-Language Model Reads From an Image Is Not What It Answers From

Look inside a vision-language model and you can often find what it saw in the image. It is natural to conclude that the model answers from what we found there. We show that it does not: what we read out is the vision encoder's description of the image, and the language model answers from somewhere else. SightLine tests use by intervention rather than by reading: we install a different value in the model's state, verify with an independent probe that it is present, and check whether the answer follows. In a controlled legend-lookup task, the probe reads the label assigned to a symbol from the symbol's own visual tokens with perfect accuracy, and the vision encoder writes it there within two blocks. A different label installed at those tokens is verified present, yet the answer never follows it: the model overwrites the installed label by mid-stack and keeps its original answer on every pair, even when the label is re-imposed at every layer. The answer depends instead on the printed label in the legend, on eight attention heads at the answer position, and on a layout-specific row identity at the symbol's own tokens that tells the model which legend row it is looking at. That identity is a handle: injected on Qwen2.5-VL-7B with no training, it switches the answer to the predicted label on 68% and 88% of held-out pairs across the two legend layouts, and back. The separation and the attention-head pathway replicate causally in three model families. On real charts the answer still depends on the printed label, the row identity is readable at legend swatches, and errors arise after a correct identity. The same pattern appears on natural benchmarks: an image-quality feature is readable in four model families, survives changes of degradation and is already present at the vision-language connector, yet predicts nothing about which items the model gets wrong. Overwriting, not reading, is the test of use. More experiments and code are provided in the appendix and https://anonymous.4open.science/r/sightline-supplement/.