- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates two different paradigms for injecting visual tokens into LLMs, both from a performance and representational/interpretability point of view: in-context injection vs. layer-wise injection. To this end they train several LLMs together with SigLIP under controlled conditions, only differing the injection paradigm. Behaviorally they find in-context, the more established paradigm, to perform better, especially on text-centric and fine-grained tasks like OCR. Next, a series of representational analysis show that: a) IN-CT allows more smooth evolution of visual representations with more modality alignment at later layers and b) that IN-CT captures more high-frequency visual features, which is also shown through causal ablations. This relates directly to tasks like OCR.
Critical strengths
Abstract is well written (and some other sections too), makes me curious to read further. Overall, really clear RQs and experiments are all easy to follow how they relate to the main RQ. The novelty is also clear, nobody has done such a controlled comparison before afaik.
The low/high frequency finding is super cool, especially if it has not been found before… Overall, the high vs. low frequency finding and its connection also to OCR-tasks is the key insight of this paper for me. The fact that much more attn goes to image tokens for some tasks (OCR) is also informative to the community.
Critical weaknesses
Related work: The paper does a decent job of distributing related work throughout the paper but since there is no related work section at all, some crucial past literature is not mentioned: “NVLM: Open Frontier-Class Multimodal LLMs” (Dai et al., 2024) seems especially relevant since it conducts this controlled comparison as well (but is of course missing the extensive analyses of the paper under review, so I don't think it is a big problem, but should be acknowledged). Some other analyses papers are more optional but also nice to acknowledge, e.g.: on low vs. high-frequency visual features, or a bit more VLM representation/interpretability work.
Some issues either in clarity or in actual arguments/results of the various analyses:
- Some CKA is quickly skimmed, not really clear how it is applied. “Sample” is just mentioned without defining it, feels like a rushed section. Similar with Fourier transform very quickly introduced.
- I am not convinced by 4.3 and find it sloppily argued: the authors make claim about semantics and how vision tokens carry rich semantic structure but PCA cannot show this. Similarly there is claims about narrow cone effects which a 3D PCA to my knowledge cannot show. I think other tools (eg LatentLens, https://arxiv.org/abs/2602.00462), could provide more direct or complimentary insights about semantic alignment in the different paradigms. It would be nice to some VLM lens results across these paradigms!
- In Figure 2 (main benchmark results): The claim that OCR and video performance stands out the most for IN-CT is not as easy to spot as the text makes it seem. Therefore, it would really help to show the avg gap in actual numbers in text. I went to appendix and also calculated the average difference and found for video it is not higher than for other task types. This should be fixed, especially since a goal of the paper is to inform architecture design for different tasks/domains.
Minor (more for improving a future revision of the paper, not influencing my judgement much):
The third paradigm, LW-AT, is not well justified, citing only 1 paper that is not a used/known model at all. I think there would have been more suitable choices for the third paradigm than this. Eg Gemma4 could be a third good paradigm to study here (training natively from scratch to process vision inside the LLM) or probably others I am not aware of. The way you motivate it later in the section as a mix of paradigm 1+2 makes more sense than just introducing it as a 3rd paradigm and citing some “random” paper.
Too many appendix references and results/plots. Less is more often, a sign that the core insight(s) have been distilled.
Questions for authors
No major questions but as already mentioned: What would a VLM lens like LatentLens show? This would make section "4.3 DO VISUAL TOKENS ALIGN WITH THE LANGUAGE REPRESENTATION SPACE?" stronger. (LogitLens could also work with less effort but only works at later layers, so not sure what to conlcude from it)
5 comments
Author Response to Reviewer 61XY (2/3)
Part 2/3
W.2.2 & Q.1 Section 4.3 PCA and VLM lens
We thank the reviewer for this valuable comment and agree that the 3D PCA alone is insufficient to show the modality gap and the convergence of visual tokens toward the language space. Following the reviewer’s comment, we now follow the original modality gap paper [3] for both the visualization and the quantitative measurement. For the visualization, we replace the 3D PCA with a 2D PCA of the image and text embeddings to show the modality gap at each layer. For the quantitative measure, we compute the modality gap as in [3], as the distance between the mean of the L2-normalized image and text embeddings (Eq. 6 in Section 4.3), at every layer of each paradigm. The results are summarized in the table below.
| Backbone | Integration | Early | Early-Mid | Mid | Late-Mid | Late |
|---|---|---|---|---|---|---|
| Qwen2.5-0.5B | L0 | L6 | L12 | L17 | L23 | |
| IN-CT | 1.26 | 0.93 | 0.98 | 0.79 | 0.38 | |
| LW-AT | 1.32 | 1.21 | 1.41 | 1.03 | 1.29 | |
| LW-GC(all) | 1.08 | 1.26 | 1.35 | 1.40 | 1.33 | |
| Qwen2.5-3B | L0 | L9 | L18 | L26 | L35 | |
| IN-CT | 1.24 | 1.03 | 0.88 | 0.74 | 0.30 | |
| LW-AT | 1.29 | 1.33 | 1.23 | 1.22 | 1.33 | |
| Qwen2.5-7B | L0 | L7 | L14 | L20 | L27 | |
| IN-CT | 1.25 | 1.19 | 1.03 | 0.93 | 0.54 | |
| LW-AT | 1.27 | 1.33 | 1.22 | 1.32 | 1.17 |
The table reports the modality gap at different depths of the LLM, where lower values indicate that the two modalities are closer and bold marks the lowest gap at each depth. The results confirm our finding in Section 4.3. The gap of IN-CT decreases with depth, from about 1.25 at the first layer to 0.30 to 0.54 at the last layer, while the gap of the layer-wise variants remains large across all layers. Additionally, the 2D PCA visualization shows the same behavior across layers, with the image and text embeddings of IN-CT visibly approaching each other as they go deeper into the LLM, while those of the layer-wise variants remain separated (Figure 6 and Figures 20 to 26 in Appendix E.4.1 of the revised paper).
Regarding the VLM lens analysis in Q.1, following the reviewer's suggestion, we apply LatentLens [4] and LogitLens [5] to individual visual tokens across layers (Figure 5 in Section 4.3, and Figures 27 to 29 in Appendix E.4.2 of the revised paper). In IN-CT, LatentLens matches each visual patch to words describing its concept, such as "seat" for a motorcycle and "jersey" for a baseball player, and these words become more specific and more similar with depth. As the reviewer pointed out, LogitLens returns uninterpretable tokens in the early to middle layers, and returns interpretable results only in the late layers. In contrast, in LW-AT and LW-GC, all patches match nearly identical words unrelated to the image, and LogitLens remains uninterpretable throughout. Together with the modality gap, these results show that IN-CT visual tokens not only move closer to the text tokens but also converge toward the semantic space of language, whereas layer-wise visual tokens remain separated from it.
In the revision, we have replaced the 3D PCA in Section 4.3 with these results and added the VLM lens analysis to Section 4.3.
W.2.3 Video result gap is not standout
We thank the reviewer for pointing this out. We have fixed this in the revision by toning down the claim and reporting the average score per task category, together with the gap to IN-CT, in Tables 2 and 3. We additionally plot this gap across model sizes in Figure 12 of the revised paper. Our intention in highlighting video was to point out an architectural disadvantage of the integration method in the layer-wise variants. As shown in [6], temporal concepts in in-context models emerge when tokens from different frames interact inside the LLM, which is only possible in IN-CT and not in the layer-wise variants, where visual tokens do not interact with each other. We have made this point clearer in the revision, explaining why IN-CT has an advantage over the layer-wise variants on video tasks.
Author Response to Reviewer 61XY (3/3)
Part 3/3
W.3 (minor) Justification about LW-AT
We thank the reviewer for this suggestion. We agree that LW-AT is not as well known as IN-CT and LW-GC. Apart from [7], this paradigm has also been adopted in other prior work, such as [8], which we discuss in Appendix A.1. We have added this citation [8] to LW-AT in Section 2.1 of the revised paper to give readers more context about this paradigm.
Regarding Gemma 4, we agree that its encoder-free design is another interesting paradigm to compare. We already discuss encoder-free approaches in Appendix A.1, and we have acknowledged Gemma 4 there in the revised paper.
We also appreciate the reviewer's point that motivating LW-AT as a mix of the two paradigms makes more sense.
W.4 (minor) Appendix
Thank you for this suggestion. Our intention with the appendix was to provide complete results and analysis for all backbones. We appreciate and agree with the reviewer’s point that the core insights should be highlighted and clearly distinguishable from the additional results. In the camera-ready version, we will reorganize the appendix to clearly separate the core results from the additional ones and highlight them.
We hope that all of our responses in this rebuttal have addressed the reviewer's concerns. If there are any remaining questions or concerns, please let us know. We would be happy to address them.
References:
[1] NVLM: Open Frontier-Class Multimodal LLMs. arXiv, 2024.
[2] What Do Self-Supervised Vision Transformers Learn? ICLR, 2023.
[3] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeurIPS, 2022.
[4] LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs. ICML, 2026.
[5] Interpreting GPT: The Logit Lens. LessWrong, 2020.
[6] Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs. ICLR, 2026.
[7] PAL: Probing Audio Encoders via LLMs, Audio Information Transfer into LLMs. arXiv, 2025.
[8] Layer-wise Vision Injection with Disentangled Attention for Efficient LVLMs. ICCV, 2025.
Author Response to Reviewer 61XY (1/3)
Part 1/3
Dear Reviewer 61XY,
We sincerely thank the reviewer for the thoughtful and constructive review, and for recognizing the novelty and key insights of our work. We truly appreciate the reviewer’s positive feedback. The reviewer’s suggestions have substantially strengthened the paper. We have uploaded a revised version of the paper, in which all changes are highlighted in orange. Please find our responses to each comment below.
W.1 Related work
We thank the reviewer for pointing us to NVLM [1], which is very closely related and valuable for our paper, and for recognizing that our analyses go beyond its benchmark-level comparison. We have acknowledged and cited NVLM in the revision (Introduction and Appendix A.2). Due to space constraints, our extended related work is provided in Appendix A, which covers integration paradigms in MLLMs (Appendix A.1) and prior analyses of MLLMs (Appendix A.2), and we have added a pointer to it in the Introduction section.
Our frequency analysis is inspired by [2], which analyzes the different visual representations obtained from different self-supervised vision models. To the best of our knowledge, our work is the first to bring this frequency perspective into VLM analysis and interpretability, where we use it to analyze how visual representations behave and change inside the LLM. For VLM interpretability, we have additionally included the VLM lens analyses, as the reviewer suggested, which we discuss in W.2.2 in Part 2/3.
W.2.1 CKA and Fourier sections
We thank the reviewer for this comment on the clarity of the methods we used. The formal definition of CKA and how we apply it are already provided in Appendix E.1. To clarify, a sample refers to one image-question pair. For each sample, we compute CKA between its visual (or text) token representations at each layer and its subsequent layers, treating each token as a data point, and average the result over all samples. This gives the CKA heatmaps shown in Figure 4.
For the Fourier analysis, we follow [2]. We arrange the visual tokens of each sample on their 24 × 24 spatial grid, apply a 2D Fourier transform to each channel, and average the amplitude over channels and samples. We then group the frequencies into 12 rings and report the relative log amplitude, defined as the difference between the log amplitude of the highest-frequency ring and that of the lowest-frequency ring. A higher value indicates more high-frequency content.
In the revised paper, we have added these definitions together with their equations to Sections 4.1 (Eq. 4) and 4.2 (Eq. 5), and the full formal description of the Fourier analysis to Appendix E.2.