acceptodds
Under review as a conference paper at ICLR 2027

The Hidden Evolution of Disguised Visual Context inside VLMs

Abstract

Visual tokens enter Large Language Models (LLMs) as raw, foreign signals. How they are transformed into consistent representations and interact with the text token space depends entirely on the integration architecture. Existing architectures typically either treat visual tokens as in-context prompts within the input sequence or inject them directly into the LLM's intermediate layers. A controlled comparison and understanding of how these architectural choices affect visual information and its internal transformation to integrate with the LLM remains underexplored. In this study, we provide a fair comparison by evaluating in-context and layer-wise injection Vision-Language Model (VLM) integration paradigms under identical training conditions on benchmarks spanning across single-image, multi-image, and video understanding. Our results show that in-context injection consistently outperforms layer-wise injection, with the largest gaps on OCR and video tasks, even though both allocate attention to visual tokens in similar patterns. This raises the question of what happens to visual information inside the LLM and how the LLM utilizes it. Investigating this, we uncover a hidden evolution of visual tokens, which enter the LLM as disguised visual context from a different representation space, lacking the structure of LLM representations, before being reshaped differently according to the integration paradigm. Our analysis reveals that this reshaping leads the visual representations in the LLM to capture fundamentally different frequency characteristics, which reflect whether they encode fine-grained local detail or coarse global structure. In-context injection progressively builds high-frequency, fine-grained detail, whereas layer-wise injection remains biased toward low-frequency features. We show that this evolution inside the LLM determines what visual features the VLM can capture and utilize effectively, such as the fine-grained detail required for text-centric tasks. It also determines whether visual representations converge toward the language space. Specifically, in-context visual tokens remain separate from text tokens through the middle layers and gradually merge with them by the final layers, whereas layer-wise visual tokens remain separated across layers. Since attention patterns are similar, the performance gap must instead come from the quality of the visual representations that each paradigm shapes at every layer. We confirm this through a causal intervention by suppressing the high-frequency content in the visual tokens inside the LLM, which degrades in-context injection across tasks, especially in text-centric ones, while layer-wise injection is far less affected. This shows that high-frequency content drives the advantage of in-context injection over layer-wise injection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Model (VLM), Visual representation, Visual integration analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_61XYOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates two different paradigms for injecting visual tokens into LLMs, both from a performance and representational/interpretability point of view: in-context injection vs. layer-wise injection. To this end they train several LLMs together with SigLIP under controlled conditions, only differing the injection paradigm. Behaviorally they find in-context, the more established paradigm, to perform better, especially on text-centric and fine-grained tasks like OCR. Next, a series of representational analysis show that: a) IN-CT allows more smooth evolution of visual representations with more modality alignment at later layers and b) that IN-CT captures more high-frequency visual features, which is also shown through causal ablations. This relates directly to tasks like OCR.

Critical strengths

Abstract is well written (and some other sections too), makes me curious to read further. Overall, really clear RQs and experiments are all easy to follow how they relate to the main RQ. The novelty is also clear, nobody has done such a controlled comparison before afaik.

The low/high frequency finding is super cool, especially if it has not been found before… Overall, the high vs. low frequency finding and its connection also to OCR-tasks is the key insight of this paper for me. The fact that much more attn goes to image tokens for some tasks (OCR) is also informative to the community.

Critical weaknesses

Related work: The paper does a decent job of distributing related work throughout the paper but since there is no related work section at all, some crucial past literature is not mentioned: “NVLM: Open Frontier-Class Multimodal LLMs” (Dai et al., 2024) seems especially relevant since it conducts this controlled comparison as well (but is of course missing the extensive analyses of the paper under review, so I don't think it is a big problem, but should be acknowledged). Some other analyses papers are more optional but also nice to acknowledge, e.g.: on low vs. high-frequency visual features, or a bit more VLM representation/interpretability work.

Some issues either in clarity or in actual arguments/results of the various analyses:

  • Some CKA is quickly skimmed, not really clear how it is applied. “Sample” is just mentioned without defining it, feels like a rushed section. Similar with Fourier transform very quickly introduced.
  • I am not convinced by 4.3 and find it sloppily argued: the authors make claim about semantics and how vision tokens carry rich semantic structure but PCA cannot show this. Similarly there is claims about narrow cone effects which a 3D PCA to my knowledge cannot show. I think other tools (eg LatentLens, https://arxiv.org/abs/2602.00462), could provide more direct or complimentary insights about semantic alignment in the different paradigms. It would be nice to some VLM lens results across these paradigms!
  • In Figure 2 (main benchmark results): The claim that OCR and video performance stands out the most for IN-CT is not as easy to spot as the text makes it seem. Therefore, it would really help to show the avg gap in actual numbers in text. I went to appendix and also calculated the average difference and found for video it is not higher than for other task types. This should be fixed, especially since a goal of the paper is to inform architecture design for different tasks/domains.

Minor (more for improving a future revision of the paper, not influencing my judgement much):

The third paradigm, LW-AT, is not well justified, citing only 1 paper that is not a used/known model at all. I think there would have been more suitable choices for the third paradigm than this. Eg Gemma4 could be a third good paradigm to study here (training natively from scratch to process vision inside the LLM) or probably others I am not aware of. The way you motivate it later in the section as a mix of paradigm 1+2 makes more sense than just introducing it as a 3rd paradigm and citing some “random” paper.

Too many appendix references and results/plots. Less is more often, a sign that the core insight(s) have been distilled.

Questions for authors

No major questions but as already mentioned: What would a VLM lens like LatentLens show? This would make section "4.3 DO VISUAL TOKENS ALIGN WITH THE LANGUAGE REPRESENTATION SPACE?" stronger. (LogitLens could also work with less effort but only works at later layers, so not sure what to conlcude from it)

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer 61XY (1/3)

Part 1/3

Dear Reviewer 61XY,

We sincerely thank the reviewer for the thoughtful and constructive review, and for recognizing the novelty and key insights of our work. We truly appreciate the reviewer’s positive feedback. The reviewer’s suggestions have substantially strengthened the paper. We have uploaded a revised version of the paper, in which all changes are highlighted in orange. Please find our responses to each comment below.


W.1 Related work

We thank the reviewer for pointing us to NVLM [1], which is very closely related and valuable for our paper, and for recognizing that our analyses go beyond its benchmark-level comparison. We have acknowledged and cited NVLM in the revision (Introduction and Appendix A.2). Due to space constraints, our extended related work is provided in Appendix A, which covers integration paradigms in MLLMs (Appendix A.1) and prior analyses of MLLMs (Appendix A.2), and we have added a pointer to it in the Introduction section.

Our frequency analysis is inspired by [2], which analyzes the different visual representations obtained from different self-supervised vision models. To the best of our knowledge, our work is the first to bring this frequency perspective into VLM analysis and interpretability, where we use it to analyze how visual representations behave and change inside the LLM. For VLM interpretability, we have additionally included the VLM lens analyses, as the reviewer suggested, which we discuss in W.2.2 in Part 2/3.


W.2.1 CKA and Fourier sections

We thank the reviewer for this comment on the clarity of the methods we used. The formal definition of CKA and how we apply it are already provided in Appendix E.1. To clarify, a sample refers to one image-question pair. For each sample, we compute CKA between its visual (or text) token representations at each layer and its subsequent layers, treating each token as a data point, and average the result over all samples. This gives the CKA heatmaps shown in Figure 4.

For the Fourier analysis, we follow [2]. We arrange the visual tokens of each sample on their 24 × 24 spatial grid, apply a 2D Fourier transform to each channel, and average the amplitude over channels and samples. We then group the frequencies into 12 rings and report the relative log amplitude, defined as the difference between the log amplitude of the highest-frequency ring and that of the lowest-frequency ring. A higher value indicates more high-frequency content.

In the revised paper, we have added these definitions together with their equations to Sections 4.1 (Eq. 4) and 4.2 (Eq. 5), and the full formal description of the Fourier analysis to Appendix E.2.


AuthorsOpenReview ↗

Author Response to Reviewer 61XY (2/3)

Part 2/3


W.2.2 & Q.1 Section 4.3 PCA and VLM lens

We thank the reviewer for this valuable comment and agree that the 3D PCA alone is insufficient to show the modality gap and the convergence of visual tokens toward the language space. Following the reviewer’s comment, we now follow the original modality gap paper [3] for both the visualization and the quantitative measurement. For the visualization, we replace the 3D PCA with a 2D PCA of the image and text embeddings to show the modality gap at each layer. For the quantitative measure, we compute the modality gap as in [3], as the distance between the mean of the L2-normalized image and text embeddings (Eq. 6 in Section 4.3), at every layer of each paradigm. The results are summarized in the table below.

BackboneIntegrationEarlyEarly-MidMidLate-MidLate
Qwen2.5-0.5BL0L6L12L17L23
IN-CT1.260.930.980.790.38
LW-AT1.321.211.411.031.29
LW-GC(all)1.081.261.351.401.33
Qwen2.5-3BL0L9L18L26L35
IN-CT1.241.030.880.740.30
LW-AT1.291.331.231.221.33
Qwen2.5-7BL0L7L14L20L27
IN-CT1.251.191.030.930.54
LW-AT1.271.331.221.321.17

The table reports the modality gap at different depths of the LLM, where lower values indicate that the two modalities are closer and bold marks the lowest gap at each depth. The results confirm our finding in Section 4.3. The gap of IN-CT decreases with depth, from about 1.25 at the first layer to 0.30 to 0.54 at the last layer, while the gap of the layer-wise variants remains large across all layers. Additionally, the 2D PCA visualization shows the same behavior across layers, with the image and text embeddings of IN-CT visibly approaching each other as they go deeper into the LLM, while those of the layer-wise variants remain separated (Figure 6 and Figures 20 to 26 in Appendix E.4.1 of the revised paper).

Regarding the VLM lens analysis in Q.1, following the reviewer's suggestion, we apply LatentLens [4] and LogitLens [5] to individual visual tokens across layers (Figure 5 in Section 4.3, and Figures 27 to 29 in Appendix E.4.2 of the revised paper). In IN-CT, LatentLens matches each visual patch to words describing its concept, such as "seat" for a motorcycle and "jersey" for a baseball player, and these words become more specific and more similar with depth. As the reviewer pointed out, LogitLens returns uninterpretable tokens in the early to middle layers, and returns interpretable results only in the late layers. In contrast, in LW-AT and LW-GC, all patches match nearly identical words unrelated to the image, and LogitLens remains uninterpretable throughout. Together with the modality gap, these results show that IN-CT visual tokens not only move closer to the text tokens but also converge toward the semantic space of language, whereas layer-wise visual tokens remain separated from it.

In the revision, we have replaced the 3D PCA in Section 4.3 with these results and added the VLM lens analysis to Section 4.3.


W.2.3 Video result gap is not standout

We thank the reviewer for pointing this out. We have fixed this in the revision by toning down the claim and reporting the average score per task category, together with the gap to IN-CT, in Tables 2 and 3. We additionally plot this gap across model sizes in Figure 12 of the revised paper. Our intention in highlighting video was to point out an architectural disadvantage of the integration method in the layer-wise variants. As shown in [6], temporal concepts in in-context models emerge when tokens from different frames interact inside the LLM, which is only possible in IN-CT and not in the layer-wise variants, where visual tokens do not interact with each other. We have made this point clearer in the revision, explaining why IN-CT has an advantage over the layer-wise variants on video tasks.


AuthorsOpenReview ↗

Author Response to Reviewer 61XY (3/3)

Part 3/3


W.3 (minor) Justification about LW-AT

We thank the reviewer for this suggestion. We agree that LW-AT is not as well known as IN-CT and LW-GC. Apart from [7], this paradigm has also been adopted in other prior work, such as [8], which we discuss in Appendix A.1. We have added this citation [8] to LW-AT in Section 2.1 of the revised paper to give readers more context about this paradigm.

Regarding Gemma 4, we agree that its encoder-free design is another interesting paradigm to compare. We already discuss encoder-free approaches in Appendix A.1, and we have acknowledged Gemma 4 there in the revised paper.

We also appreciate the reviewer's point that motivating LW-AT as a mix of the two paradigms makes more sense.


W.4 (minor) Appendix

Thank you for this suggestion. Our intention with the appendix was to provide complete results and analysis for all backbones. We appreciate and agree with the reviewer’s point that the core insights should be highlighted and clearly distinguishable from the additional results. In the camera-ready version, we will reorganize the appendix to clearly separate the core results from the additional ones and highlight them.


We hope that all of our responses in this rebuttal have addressed the reviewer's concerns. If there are any remaining questions or concerns, please let us know. We would be happy to address them.


References:

[1] NVLM: Open Frontier-Class Multimodal LLMs. arXiv, 2024.

[2] What Do Self-Supervised Vision Transformers Learn? ICLR, 2023.

[3] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeurIPS, 2022.

[4] LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs. ICML, 2026.

[5] Interpreting GPT: The Logit Lens. LessWrong, 2020.

[6] Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs. ICLR, 2026.

[7] PAL: Probing Audio Encoders via LLMs, Audio Information Transfer into LLMs. arXiv, 2025.

[8] Layer-wise Vision Injection with Disentangled Attention for Efficient LVLMs. ICCV, 2025.

Reviewer_ToWsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares the impact of different visual token injection methods, including in-context (LLaVA-style), LWGC (Flamingo-style), and LWAT (PaL-style), with Qwen 2.5 and LLaMA-3 backbones across single-image, multi-image, and video benchmarks. The authors find that the in-context architecture generally performs better and that different architectures exhibit task-oriented preferences. To explain the performance gaps among the methods, they analyze layer-wise visual representation changes, apply Fourier analysis to fine-grained and coarse-grained features, use layer-wise PCA to examine modality gaps, evaluate attention locations, and investigate the effect of suppressing high-frequency visual content.

Critical strengths
  1. Well-controlled comparison: The encoder, connector, backbone, training data, and hyperparameters are all held constant. All models are fine-tuned end-to-end, making a direct architectural comparison possible.
  2. Comprehensive benchmarks: The evaluation spans approximately 20 benchmarks, including single-image, multi-image, and video tasks, and providing comprehensive results to compare the three architectures.
  3. Good analysis: The paper analyzes multiple perspectives to explain why the in-context approach generally performs best across all experiments.
Critical weaknesses
  1. Although the controlled comparison experiments are well-designed, most of the findings lack novelty. For example, VISOR [1] uses layer-wise CKA heatmaps to inspect visual representations in the LLaVA model and finds that cross-attention does not update visual tokens significantly, consistent with CASA [2]; NVLM [3] also compares decoder-only and cross-attention designs.
  2. The definition of "high-frequency" in this paper is unclear.
  3. The claim that text-centric tasks have higher amplitudes holds only for the 0.5B model. In the appendix, ChartQA has the lowest values across most layers for the 3B model, and at 7B, COCO and RealWorldQA perform better in the middle layers.
  4. There’s already some papers about the VLMs is sensitive to the task types. Seeing or knowing [4], Pathways[5].
  5. For the layer-wise representation comparisons, the observed differences are not surprising. Adding raw representations from the visual encoder directly to the middle layers of an LLM naturally alters the CKA (see Weakness 1).
  6. The evaluated LLM architectures are outdated. Qwen3-VL uses the DeepStack architecture, Qwen 3.5 uses hybrid attention, and Gemma 4 uses per-layer embeddings. Are these analyses and findings still applicable to newer open-weight models?

[1]Bulat, Adrian, et al. "VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions." [2]Böhle, Moritz, et al. "CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion." [3]Dai, Wenliang, et al. "Nvlm: Open frontier-class multimodal llms." [4]Li, Jiaang, et al. "Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models." [5]Salazar, Israfel, et al. "Pathways of Visual Information Flow in Vision-Language Models.".

Questions for authors
  1. How to explain the order changes in figure 12 and figure 13?
  2. For ablation studies, when combine two methods together, will it have more visual tokens comparing to only one method? Does the improvement come from more tokens or truely come from the combination?
  3. Is there any scaling figures to show the performance gap across different model size? Will the gap become larger or smaller?
  4. If I understand correctly, a single high-norm outlier token can shift the gaps in figure 4 (d-f). Is there any analysis for the visual attention sink?
8 comments
AuthorsOpenReview ↗

Author Response to Reviewer ToWs (1/6)

Part 1/6

Dear Reviewer ToWs,

We thank the reviewer for the time spent reviewing our paper and for the constructive feedback. The reviewer's comments have helped us better position our study against related work and extend our analyses to recent VLMs and visual attention sinks, which strengthened our paper.

We have uploaded a revised version of the paper, in which all changes are highlighted in orange and the title has been revised to "How Should Vision Enter the Language Model? A Comparative Study of Vision Integration Paradigms", as suggested by Reviewer 61XY. Please find our responses to each comment below.


W.1 & W.4 Novelty and related work (1/2)

We thank the reviewer for pointing us to these works, and we have cited and discussed all of them in the revised paper. We agree that these works are closely related. Two of them are prior works, and the other three are concurrent works that became publicly available after our study was first submitted to another conference. We clarify below how our study differs from them.

1. Prior works that compare integration designs at the benchmark level, NVLM [3] and CASA [2].

NVLM [3] compares decoder-only, cross-attention, and hybrid designs with a frozen vision encoder to build a frontier-class VLM, and CASA [2] proposes a more efficient cross-attention design and compares it with token insertion. Our work is complementary, since we also study the differences between integration paradigms of VLMs. However, our study differs in two ways.

  • Training setup. We fine-tune the vision encoder, projector, and LLM end-to-end for all three paradigms under the same controlled setup, following modern VLM training practice, whereas NVLM keeps the vision encoder frozen and CASA fine-tunes only its last four layers.
  • Analysis beyond benchmarks. While both studies focus on efficiency and benchmark performance, our study goes beyond benchmark numbers, as we conduct an extensive analysis of how the integration paradigms differ inside the LLM (Section 4). We also identify one of the key reasons why layer-wise variants fall behind IN-CT on text-centric tasks, as they lack the high-frequency features that IN-CT builds.

2. Concurrent works that analyze visual processing inside in-context models, VISOR [1], Seeing or Knowing [4], and Pathways [5].

First, VISOR [1] also applies CKA on an in-context model and observes that visual tokens change more across layers on fine-grained tasks, which is consistent with our observation for IN-CT. However, while VISOR uses CKA on a single in-context model to compare their easy and hard tasks, our study systematically compares three integration paradigms trained under identical conditions through several analyses, and goes beyond VISOR in three ways.

  • Comparison across paradigms. We apply CKA across three paradigms and show that in-context visual tokens evolve smoothly across layers, while layer-wise injection produces discontinuous visual representations (Section 4.1).

  • What the visual tokens encode. Beyond how much visual tokens change, we also show what this change consists of. IN-CT visual tokens build high-frequency content (Fourier analysis, Section 4.2), and converge toward the language space (modality gap and VLM lens, Section 4.3), while layer-wise visual tokens do neither.

  • Effect on performance. We show that IN-CT benefits from text-centric training data, which is rich in high-frequency content, while the layer-wise variants do not (Section 3.1). Through a causal intervention, we further show that suppressing this content degrades IN-CT mostly on text-centric tasks, showing that it is one of the key contributors to the advantage of IN-CT (Section 4.5).


AuthorsOpenReview ↗

Author Response to Reviewer ToWs (2/6)

Part 2/6


W.1 & W.4 Novelty and related work (2/2, continued from Part 1/6)

Secondly, Seeing or Knowing [4] shows that MLLMs encode visual evidence but cannot reliably control how much they rely on it, and that this reliance varies across attribute types. Pathways [5] shows that the pathway through which visual information flows inside the LLM depends on the task. Both studies analyze only in-context models, and their findings are consistent with our observation that IN-CT uses visual information differently across tasks. However, our study goes beyond these two studies in three ways.

  • Text-centric tasks. Neither of these two studies covers text-centric tasks such as OCR, where we find the largest differences between integration paradigms.
  • Attention across paradigms. We show that this task dependence also holds for LW-AT, as both IN-CT and LW-AT attend to visual tokens in the same task-dependent way (Section 4.4), so attention alone does not explain the performance gap.
  • Visual representations. We identify that this gap also reflects the visual representations. IN-CT builds more high-frequency content for text-centric inputs than for general inputs, while the layer-wise variants do not (Section 4.2), and removing this content degrades IN-CT mostly on text-centric tasks (Section 4.5).

Timeline of our study. To clarify the timeline relative to these concurrent works, the following core analyses of our study were already part of our first submission.

  1. A controlled comparison of IN-CT, LW-AT, and LW-GC under identical training conditions with Qwen2.5-0.5B and 3B (Sections 2 and 3)
  2. Representation evolution of visual tokens across layers with CKA (Section 4.1)
  3. Frequency content of visual tokens with Fourier analysis (Section 4.2)
  4. Convergence of visual tokens toward the language space (Section 4.3)
  5. Attention mass on visual tokens during generation (Section 4.4)

The version under review further extends this study with the following additions.

  1. A causal intervention that suppresses high-frequency content (Section 4.5)
  2. A larger model scale, Qwen2.5-7B, and a second LLM family, LLaMA-3.2-1B and 3B (Section 3)

During this rebuttal, we have further added the following analyses to the revised paper.

  1. VLM lens analysis with LatentLens and LogitLens (Section 4.3 and Appendix E.4.2), following Reviewer 61XY (please see W.2.2 & Q.1 in our response to Reviewer 61XY)
  2. 2D PCA with the modality gap measured at each layer (Section 4.3 and Appendix E.4.1), following Reviewer 61XY (please see W.2.2 & Q.1 in our response to Reviewer 61XY)
  3. Analysis on recent VLMs, Qwen3-VL, Qwen3.5, and Gemma-4 (Appendix F), following the reviewer's W.6
  4. Robustness to visual attention sinks (Appendix E.5), following the reviewer's Q.4
  5. Longer training of the layer-wise variants (Appendix D.4), following Reviewer Kqu4 (please see W.2.1 in our response to Reviewer Kqu4)

We hope this clarifies that the core analyses of our study were developed independently of these concurrent works. If needed, we are happy to share the name of the conference and our submission number with the Area Chair in a confidential comment, so that the timeline can be verified without breaking anonymity.

We thank the reviewer again for pointing us to these works, which gave us the opportunity to better position our paper, clarify its contributions, and clarify its relation to concurrent work. We have added a discussion of them in Appendix A.2 of the revised paper, including how our study differs from the concurrent works, and we will further expand this discussion in the camera-ready version.


AuthorsOpenReview ↗

Author Response to Reviewer ToWs (3/6)

Part 3/6


W.2 The definition of high-frequency is unclear

We thank the reviewer for this comment and have made the definition explicit in the revision.

  • Definition. In our analysis, high-frequency content refers to variations between neighboring patches, which arise from fine-grained local details such as edges, textures, and text strokes. Images in OCR & Chart tasks mainly contain such fine-grained details, and therefore rely heavily on high-frequency content. Low-frequency content refers to smooth variations across the image, which capture coarse global structure.
  • Measurement (Section 4.2). We arrange the visual tokens at each LLM layer on their 24 × 24 patch grid, apply a 2D Fourier transform to each channel of this map, average the amplitude over channels and samples, group the frequencies into rings by their distance from the center, and report the relative log amplitude, the difference between the log amplitude of the highest- and lowest-frequency rings. A higher value indicates more high-frequency content.

With this measure, we show that IN-CT captures more high-frequency content than the layer-wise variants (Section 4.2), and that removing it mainly degrades IN-CT on OCR & Chart tasks (Section 4.5). We have added this definition together with its equation (Eq. 5) to Section 4.2 and the full formal description to Appendix E.2 of the revised paper.


W.5 The layer-wise CKA differences are not surprising

We agree that injecting encoder features at intermediate layers can be expected to change the CKA of the visual tokens, and we already acknowledge this in Section 4.1, where we state that "while different LLM layers may require different interpretations of visual features, these results show that the two integration families arrive at fundamentally different solutions to this problem." The purpose of the CKA analysis is to show how differently the two families handle it, with smooth evolution in IN-CT but discontinuities in LW-AT and LW-GC. To the best of our knowledge, this has not been confirmed before under controlled conditions, where the vision encoder and training data are the same and only the integration method differs. Our results show that the integration method alone leads to very different visual representations, and this observation is the starting point of our analysis rather than its conclusion. It further led us to the analyses of visual frequency content (Section 4.2) and alignment toward the language space (Section 4.3).


Q.2 Ablation studies on Hybrid models

We thank the reviewer for this question. Yes, the hybrid model uses twice as many visual tokens as IN-CT, as we note in Section 5, where we describe it as computationally impractical and present it as an exploratory finding. To prove that its improvement does not simply come from the larger number of visual tokens, we additionally train IN-CT with two projectors. The same vision encoder features are passed to both projectors, and each projector produces the same number of visual tokens as the single projector in IN-CT. Together, they provide twice as many visual tokens, matching the hybrid model, while everything else is kept the same. The two sets of visual tokens are placed in the same token order as in the hybrid model (Figure 11 in Appendix B.4), so the only difference is that both sets are integrated in context rather than through two different paradigms.

TaskHybridIN-CT (two projectors)Gap
MMB54.153.5−0.6
GQA57.857.5−0.3
ChartQA48.744.3−4.4
TextVQA48.847.3−1.5
DocVQA38.638.0−0.6
OCRBench41.137.4−3.7
RealWorldQA51.449.7−1.7
CV-Bench 2D44.445.0+0.6
CV-Bench 3D51.650.0−1.6
MMMU31.331.1−0.2
ScienceQA60.460.3−0.1
Average48.046.7−1.3

As shown in the table above, IN-CT with two projectors is 1.3 points lower than the hybrid model on average over these 11 benchmarks, so more visual tokens alone do not lead to better performance. This shows that the improvement of the hybrid model comes from combining the different features provided by the two paradigms, which are the high-frequency features of IN-CT and the low-frequency features of LW-AT shown in Section 4.2.


Reviewer_Kqu4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares in-context and layer-wise visual injection in VLMs under a shared training recipe. In-context injection generally performs better, particularly on OCR and video tasks. Through representation, frequency, and attention analyses, together with low-pass interventions, the authors attribute this advantage to the progressive refinement of visual representations inside the LLM.

Critical strengths
  1. Useful architectural comparison. The experiments cover two backbone families, multiple model scales, and single-image, multi-image, and video benchmarks, providing a broad comparison under matched data and training recipes.

  2. Complementary analyses of visual processing. The paper combines representation analysis with targeted frequency interventions, connecting architectural choices to both internal behavior and downstream performance.

Critical weaknesses
  1. The causal interpretation exceeds what the intervention establishes. Low-pass filtering shows that IN-CT is more sensitive to high-frequency removal, but does not establish that this explains the architectural performance gap. Moreover, the claim of similar attention allocation is less evident in the 3B results in Figure 15.

  2. Some experimental effects remain entangled. Matching epochs does not match computation, leaving the performance–efficiency trade-off unclear. Removing text-centric data also reduces the training set from approximately 700K to 500K, so the observed changes cannot be attributed solely to data type.

Questions for authors

The questions of what visual tokens already encode when entering the LLM and what further processing inside the LLM contributes have also been explored in prior work, such as [1][2]. Although the methods differ, could the authors discuss whether the two studies share similar observations and whether their findings support each other?

[1] What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models. CVPR 2026.
[2] Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations,ICLR 2025.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer Kqu4 (1/3)

Part 1/3

Dear Reviewer Kqu4,

We thank the reviewer for the time spent reviewing our paper and for the positive feedback. We are glad that the reviewer found our comparison useful and our analyses complementary, and the reviewer's comments have helped us clarify the scope of our claims, which improved our paper. We have uploaded a revised version of the paper, in which all changes are highlighted in orange and the title has been revised to “How Should Vision Enter the Language Model? A Comparative Study of Vision Integration Paradigms”, as suggested by Reviewer 61XY. We address each of the reviewer's comments below.


W.1.1 Low-pass filtering does not establish that this explains the architectural performance gap

We thank the reviewer for raising this point. We agree that the low-pass intervention on high-frequency content may not explain the full gap between IN-CT and the layer-wise variants. Therefore, we have narrowed our claim in the revised paper. Our claim is that high-frequency content is one of the key contributors to the advantage of IN-CT, though not necessarily the only one. This claim is supported by several results and analyses across the paper.

  1. The gap is largest on text-centric tasks, which are rich in high-frequency content. IN-CT outperforms the layer-wise variants across benchmarks, with the largest gap on OCR & Chart tasks (Section 3). These tasks require the model to recognize text, numbers, and thin lines in images, which are fine-grained details corresponding to high spatial frequencies.

  2. IN-CT benefits more from data rich in high-frequency content. Removing text-centric training data, which is rich in fine-grained details, degrades IN-CT on nearly every benchmark, while the layer-wise variants are less affected and even improve on several benchmarks (Section 3.1). This suggests that IN-CT relies on and benefits more from high-frequency content than the layer-wise variants do.

  3. IN-CT captures more high-frequency content. Our Fourier analysis shows that IN-CT captures substantially more high-frequency content than the layer-wise variants, especially for text-centric inputs (Section 4.2).

  4. Suppressing high-frequency content hurts IN-CT the most. Suppressing high-frequency content in the visual tokens inside the LLM degrades IN-CT mostly on OCR & Chart tasks, and much less on General Knowledge tasks, while the same intervention affects the layer-wise variants far less (Section 4.5).

Together, these results indicate that the lack of high-frequency content is one of the reasons the layer-wise variants underperform IN-CT, especially on text-centric tasks. To make this scope explicit, we have revised the abstract, introduction, Section 4, Section 4.5, Finding 5, and the conclusion to narrow this claim, no longer stating that high-frequency content alone drives the advantage of IN-CT, but that it is one of the key contributors.


W.1.2 The claim of similar attention allocation is less evident in the 3B results

We agree that the attention allocation of IN-CT and LW-AT is less similar at 3B than at 0.5B (Figure 15 in the original submission, Figure 19 in the revised paper). However, at both scales, both paradigms still show the same task-dependent pattern, where OCR & Chart tasks allocate more attention to image tokens in the middle and later layers than non-OCR tasks, which is the main point of our claim. In the revised paper, we have toned down this claim throughout the paper, now stating that the two paradigms share the same task-dependent attention pattern rather than allocating attention similarly.


AuthorsOpenReview ↗

Author Response to Reviewer Kqu4 (2/3)

Part 2/3


W.2.1 Matching epochs does not match computation

We thank the reviewer for pointing this out. We agree that matching epochs only ensures that all paradigms are trained on the same data budget, and does not equalize training compute. As we had already acknowledged this limitation in our Limitations section (Section 6), we had prepared additional experiments in advance, in which LW-AT and LW-GC(all) are trained for two epochs to provide a closer approximation to the training compute of IN-CT. A summary of the results is provided in the table below, and the full results are provided in Appendix D.4 of the revised paper, with the category averages in Table 6, the per-benchmark results in Table 7, and the gap across model sizes in Figure 13. We have also updated the Limitations section to refer to these results.

BackboneIN-CT (1 ep)LW-AT (1 ep)LW-AT (2 ep)LW-GC(all) (1 ep)LW-GC(all) (2 ep)
Qwen2.5-0.5B45.341.041.432.533.1
Qwen2.5-3B55.047.448.4––
Qwen2.5-7B58.652.453.4––
LLaMA-3.2-1B46.843.144.735.737.1
LLaMA-3.2-3B53.747.649.1––

The table reports the average performance over all 20 benchmarks. The layer-wise variants improve with a larger training budget, but remain clearly behind IN-CT even with twice their training budget. Moreover, LW-AT trained for two epochs shows nearly the same internal behavior as when trained for one epoch, in terms of CKA, frequency content, and modality gap (Figure 14 in the revised paper).


W.2.2 Removing text-centric data also reduces the training set

We agree that removing text-centric data would reduce the training set size, and this would be expected to affect all paradigms in the same direction equally. However, the impact of this text-centric filter is much larger on IN-CT, which degrades on nearly every benchmark, while the layer-wise variants are less affected and even improve on several benchmarks (Figure 3 and Table 4). As shown in Figure 3(b), IN-CT drops more on average, whereas the layer-wise variants drop less or even improve. This difference suggests that the effect comes from the type of data that each integration paradigm can benefit from during training, rather than from the amount of data. Since text-centric data is rich in high-frequency content, this is consistent with our finding that IN-CT relies on and benefits more from high-frequency content than the layer-wise variants (see W.1.1).


AuthorsOpenReview ↗

Author Response to Reviewer Kqu4 (3/3)

Part 3/3


Q.1 Share similar observations and whether their findings support each other?

We thank the reviewer for pointing us to these valuable works [1] and [2]. Both studies are largely consistent with our findings and complementary to them.

  1. First, [1] uses EmbedLens to study what the visual tokens of in-context models encode before and inside the LLM, and its findings are consistent with ours in two ways.

    • Visual attention sinks (Appendix E.5). [1] shows that only part of the visual tokens carry image-specific meaning, and that removing sink tokens causes no degradation. This is consistent with our analysis of visual attention sinks, which we added to the revised paper following the request of Reviewer ToWs (see Q.4 in our response to Reviewer ToWs), where the sink tokens are inherited from the vision encoder and do not change our findings.

    • Processing inside the LLM (Sections 4.2 and 4.5). The experimental results of [1] also show that OCR performance drops when either the self-attention or the FFN update of the visual tokens inside the LLM is removed. This is consistent with our finding that IN-CT builds the high-frequency content needed for text-centric tasks through its processing inside the LLM.

    While [1] studies IN-CT models only, our analysis extends these observations across integration paradigms. The sink tokens appear in both IN-CT and LW-AT, showing that they do not depend on the integration paradigm, whereas the high-frequency content needed for text-centric tasks is built only in IN-CT, where the visual tokens are processed in context by the LLM.

  2. Secondly, [2] projects the internal image representations of in-context models onto the language vocabulary using the logit lens [4], and finds that these representations give more confident probabilities to objects present in the image than to hallucinated ones, which can also be used to localize these objects in the image. This is consistent with our findings in two ways.

    • Finding 3. The modality gap shows that IN-CT visual tokens converge toward the language space as they go deeper into the LLM.

    • VLM lens (Figure 5). Following the suggestion of Reviewer 61XY (see W.2.2 & Q.1 in our response to Reviewer 61XY), we have additionally applied LatentLens [3] and LogitLens [4] in the revised paper. Consistent with [2], LatentLens matches IN-CT visual tokens to words describing the image patch, and LogitLens returns these words in the final layer.

    While [2] studies IN-CT models only, our analysis extends this observation across integration paradigms. Under the layer-wise variants, we observe none of these behaviors, showing that this property is not a general property of VLMs but depends on the integration paradigm, emerging only when visual tokens are processed in context by the LLM.

We thank the reviewer again for this question, which gave us the opportunity to discuss how our findings relate to closely related work. [2] is already cited in our paper. We also thank the reviewer for bringing [1] to our attention, and we have cited it and included this discussion in Appendix A.2 of the revised paper.


We hope that all of our responses in this rebuttal have addressed the reviewer's concerns. If there are any remaining questions or concerns, please let us know. We would be happy to address them.


References:

[1] What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models. CVPR, 2026.

[2] Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations. ICLR, 2025.

[3] LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs. ICML, 2026.

[4] Interpreting GPT: The Logit Lens. LessWrong, 2020.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thank you reviewer for title change suggestion

We thank the reviewer for this helpful suggestion, and we are very glad that the reviewer enjoyed the insights of our paper. We agree that the current title is too generic and does not convey what the paper studies. Following the reviewer's suggestion, we have updated the title in the revised paper to "How Should Vision Enter the Language Model? A Comparative Study of Vision Integration Paradigms", which directly reflects our controlled comparison of in-context and layer-wise injection. We have uploaded the revised paper with the updated title.

AuthorsOpenReview ↗

Summary of Key Contributions and Rebuttal

We sincerely thank the AC and SAC for managing the review process and the reviewers for their valuable feedback. We are especially grateful to Reviewers ToWs and 61XY for actively engaging in the discussion.

Summary of Contributions

  • Controlled Comparison: We provide a controlled comparison of in-context (IN-CT) and layer-wise (LW-AT, LW-GC) vision integration, where only the integration paradigm differs while the vision encoder, training data, and training recipe are kept the same, across two LLM families and model sizes from 0.5B to 7B.

  • Benchmark Findings: IN-CT consistently performs best across single-image, multi-image, and video benchmarks, with the largest gap on text-centric tasks such as OCR & Chart.

  • Analysis Findings: We further analyze how different paradigms reshape the visual representations inside the LLM, which helps explain the performance gap and offers insights for future VLM design. Our main analyses and findings are as follows.

    • Representation evolution. IN-CT visual tokens evolve smoothly across layers, while layer-wise visual tokens change discontinuously.

    • Frequency content. IN-CT builds high-frequency content, while layer-wise visual tokens remain biased toward low frequencies.

    • Alignment with the language space. IN-CT visual tokens converge toward the language space, while layer-wise visual tokens remain separated from it.

    • Attention. Both paradigms share the same task-dependent attention pattern, so attention alone does not explain the performance gap.

    • Causal intervention. Our causal intervention suppresses high-frequency content inside the LLM. This affects IN-CT the most, especially on text-centric tasks. This shows that high-frequency content is one of the key contributors to the advantage of IN-CT.

Reviewer Responses

  • Kqu4 (Original Score 3, response pending). Concerned about the scope of the claim and fairness in training compute of layer-wise variants. We clarified the claim and provided results of layer-wise variants with longer training.

  • ToWs (Score increased from 2 to 3). Concerned about novelty and the robustness of our findings. We clarified our differences from prior and concurrent work and added further analyses to support our findings.

  • 61XY (Original Score 3). Concerned about related work, the clarity of our methods, and the support for some of our claims. We added the related work, clarified our methods, and supported these claims with additional results. We also revised the title, as suggested by the reviewer. The reviewer stated that the paper should be accepted and that they are between 3 and 4, with the score maintained at 3.

We believe the main reviewer concerns have been addressed, and we will include all new experiments and discussed works in the camera-ready version. We trust the AC will recognise the value of our study and its insights for designing future VLMs.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.