acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Current Answer: Visual Content Transfer into VLM Text States

Abstract

When a vision–language model tolerates removal of visual states, has it retained image content or only computed the current answer? We separate three questions: which states control the answer, whether text states contain information beyond that answer, and whether visual states remain necessary. Our central test pairs real images with identical questions and annotated answers but different unasked object categories. Patching donor visual activations early versus late changes which image's unasked content is decodable from non-final text states. Across seven tested checkpoints and 500 disjoint pairs per checkpoint, the paired interactions are positive, including a model with no sustained visual–text deletion gap. Complementary opposite-answer patches reveal a depth-dependent shift of causal control from visual to non-final text states. Symmetric zero-out on 1,000 ScienceQA questions yields a positive first-recovery gap in six of seven checkpoints, but sustained ordering is architecture-dependent; open-ended TextVQA reverses the first-recovery ordering in two models. Position-controlled probes expose a substantial causal-order confound. These results support transfer of image-derived semantic content beyond the current answer into text residuals. They do not establish lossless image storage or unrestricted re-queryability: a visual-KV-blocked second-question test largely fails. Content transfer and visual-token dispensability are distinct properties.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bXeyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether visual information in VLMs is transferred into text hidden states, rather than only being used to compute the current answer. The authors use answer-matched image pairs and activation patching to test whether text states contain information about unasked objects. They also examine how visual and text states influence the generated answer and how model performance changes when these states are zeroed out. The results suggest that text states can carry visual information beyond the current answer. However, this does not necessarily mean that visual tokens can be safely removed or that the model can use this information to answer new questions.

Critical strengths
  1. The paper addresses an interesting question by distinguishing visual information transfer from simply computing the current answer. It also makes a useful distinction between content transfer, answer control, and visual token removal.

  2. The answer-matched donor patching experiment is well motivated. By keeping the question and answer fixed while changing other objects in the image, it provides evidence that text states contain information beyond the current answer. The results across multiple models and additional controls strengthen this finding.

  3. The paper also reports negative results, such as the reversed recovery order on TextVQA and the failure of the second-question experiment. These results help clarify the limitations of visual-to-text transfer.

Critical weaknesses
  1. In [1], they have already studied cross-modal information flow, while [2] showed that query tokens can carry image information. The answer-matched donor experiment provides additional evidence, but the paper does not offer much new understanding of how this transfer happens.

  2. The main experiment focuses on object presence using COCO annotations and a trained probe. This shows that some object information can be decoded from text states, but it is unclear how much other visual information is retained. The second-question experiment also shows that the VLM cannot easily use this information to answer new questions.

  3. Visual and text tokens have different attention contexts, so the stronger effect of text patching may partly come from their positions rather than information transfer alone. The authors discuss this issue, but it still limits the interpretation of the carrier handoff.

[1] Zhang, Z., Yadav, S., Han, F., and Shutova, E. (2025). Cross-modal Information Flow in Multimodal Large Language Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19781–19791. https://openaccess.thecvf.com/content/CVPR2025/html/Zhang_Cross-modal_Information_Flow_in_Multimodal_Large_Language_Models_CVPR_2025_paper.html

[2] Kaduri, O., Bagon, S., and Dekel, T. (2025). What's in the Image? A Deep-Dive into the Vision of Vision Language Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14549–14558. https://openaccess.thecvf.com/content/CVPR2025/html/Kaduri_Whats_in_the_Image_A_Deep-Dive_into_the_Vision_of_CVPR_2025_paper.html

Questions for authors
  1. In the donor patching experiment, the late-patched text states are read immediately after the same block. Would the target-specific information still be preserved if the readout were taken several layers later?

  2. Compared with [1] and [2], what do the authors consider the main new insight of this work beyond providing additional evidence for visual-to-text information transfer?

  3. [2] showed that query tokens can support image description without direct access to visual tokens. However, the second-question experiment in this paper largely fails. What do the authors think causes this difference? How should we understand the gap between information being decodable by a probe and being usable by the VLM itself?

[1] Zhang, Z., Yadav, S., Han, F., and Shutova, E. (2025). Cross-modal Information Flow in Multimodal Large Language Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19781–19791. https://openaccess.thecvf.com/content/CVPR2025/html/Zhang_Cross-modal_Information_Flow_in_Multimodal_Large_Language_Models_CVPR_2025_paper.html

[2] Kaduri, O., Bagon, S., and Dekel, T. (2025). What's in the Image? A Deep-Dive into the Vision of Vision Language Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14549–14558. https://openaccess.thecvf.com/content/CVPR2025/html/Kaduri_Whats_in_the_Image_A_Deep-Dive_into_the_Vision_of_CVPR_2025_paper.html

Reviewer_aHx6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates information transfer from visual tokens into text states in vision–language models, focusing on whether these states retain image content beyond the current answer. The authors use answer-matched image pairs, activation patching, and probing to study unasked object information in non-final text states. The paper presents empirical analyses across multiple VLM checkpoints and benchmarks, examining the relationship between visual information retention, causal answer control, and visual-token dispensability.

Critical strengths
  1. Understanding how vision–language models process and transfer visual information is an interesting research problem.
  2. The answer-matched pairing design provides a useful control for examining information beyond the queried answer. Holding the question and annotated answer fixed while varying unasked object categories helps distinguish broader image-related information from representations of the current answer.
Critical weaknesses
  1. The contribution beyond prior work is limited. Previous studies [1, 2] have already investigated visual-to-text information flow and the retention of visual information in question-token representations. The answer-matched pairing design provides an additional diagnostic control, but the paper does not clearly establish a substantially new mechanism or understanding of VLM behavior beyond these existing findings.

  2. The experiments do not sufficiently isolate the central claimed dissociation. Content transfer is primarily evaluated on COCO, whereas deletion behavior is examined on ScienceQA and TextVQA, using different token masks and intervention definitions. Consequently, task differences and intervention choices remain alternative explanations for the combined observations. Moreover, zeroing visual residual states leaves earlier visual KV routes available, limiting what recovery establishes about visual-token dispensability. A comparison on the same examples with matched interventions is needed to support a stronger mechanistic conclusion.

  3. The connection between decodable information and functional use remains unclear. The probes and donor-patching experiments support the presence of image-dependent information about unasked categories in text states, but do not show that the VLM uses these variables in its own predictions. The separate carrier-patching experiment concerns the current answer and therefore does not close this gap.

  4. Some figures are difficult to read. In particular, the legend in Figure 5 overlaps with the plotted curves, obscuring the results. The figure layout should be revised to improve readability.

[1] Zhang, Zhi, et al. "Cross-modal information flow in multimodal large language models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [2] Kaduri, Omri, Shai Bagon, and Tali Dekel. "What’s in the Imageƒ A Deep-Dive into the Vision of Vision Language Models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors
  1. What new mechanistic insight does the answer-matched counterfactual experiment establish beyond the visual-to-text information transfer reported existing works [1, 2]?

  2. Can you provide evidence connecting the unasked information decoded by the probes to its functional use by the VLM? How should the positive probing results be interpreted alongside the largely unsuccessful second-question experiment?

  3. Does the dissociation between content transfer and deletion tolerance persist when both are measured on the same image-question cohort? How sensitive is the recovery ordering to retained visual KV access and the preserved final text positions?

[1] Zhang, Zhi, et al. "Cross-modal information flow in multimodal large language models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [2] Kaduri, Omri, Shai Bagon, and Tali Dekel. "What’s in the Imageƒ A Deep-Dive into the Vision of Vision Language Models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Reviewer_J4wPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether VLMs transfer visual information beyond the current answer into text hidden states. Using answer-matched image pairs, activation patching, and linear probes, the authors show that non-final text states encode information about unasked object categories across seven VLM checkpoints. Additional experiments examine how causal control over the current answer shifts from visual to text states across layers, and how the effects of visual-state removal vary across models and tasks.

Critical strengths

The experiments cover multiple VLMs and include extensive controls and robustness checks.

Critical weaknesses
  • Unclear novelty and significance. My main concern is that the paper does not sufficiently establish why its central finding is important or what new understanding it provides to the community. Previous studies have already shown that visual information, including general scene-level information, is transferred into question-token representations. Given the causal attention structure of decoder-only VLMs, it also seems reasonable to expect that text hidden states may contain visual information beyond what is strictly necessary for the current answer. it is unclear what fundamentally new insight this work provides, beyond a more controlled characterization of an already observed behavior.
  • Poor clarity and presentation. The paper is very difficult to follow. Sections 2.2 and 2.3 introduce complicated experimental setups, interventions, and evaluation metrics without sufficient intuitive explanations. More importantly, the high-level motivation and connections between experiments are not clearly communicated.
  • Limited mechanistic and functional insights. The paper demonstrates that additional visual information is decodable from text states, but provides limited insight into how this information is transferred or whether it plays a functional role in downstream reasoning. The negative second-question results further raise questions about why the transferred information cannot be effectively reused and what implications the central finding has for understanding VLM computation.
Questions for authors

What is the broader motivation and significance of studying whether text hidden states encode visual information beyond the current answer? While the paper provides evidence for this phenomenon, it remains unclear what new insights this finding offers into VLM computation, or why it is important for the research community to understand it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.