acceptodds
Under review as a conference paper at ICLR 2027

Encoder-Free Multimodal LLMs Learn Implicit Encoders

Abstract

We empirically analyze two recently proposed encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small. Unlike previous multimodal LLMs, these models do not use modality-specific encoders to extract tokenized embeddings representing audio and image input; instead, they directly feed windows of samples, pixels, or spectral features to the Transformer language model. We analyze whether the early layers of these models learn to function similarly to dedicated audio and visual encoders. We find that these early layers do in fact act as “implicit encoders,” encoding audio and visual inputs in a way that is largely invariant to their cross-modal context. Furthermore, when performing multimodal reasoning tasks, we show that independently encoding the modalities throughout these early model layers achieves the same accuracy as jointly encoding them from the first layer onwards. Finally, we use layer-wise probing to analyze the salient information captured by the implicit encoders and find their behavior to be similar to several popular encoder models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal LLMs, multimodal encoders

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z8r3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper anlayze whether the early layers of encoder-free multimodal LLMs function similary as the dedicated audio and visual encoders. The extensive results shows the evidence of these early layers act as implicit encoders.

Critical strengths

Well-motivated study of a timely and relatively underexplored problem in encoder-free multimodal LLMs. The experimental design is well structured and comprehensive.

Critical weaknesses
  1. Section 3.4 states that the visual representations of both Gemma-4 12B and Inkling-Small are evaluated as vision encoders, yet the reported MIEB results and the appendix only include Gemma-4 12B.
  2. Section 3.1.1 evaluates context invariance only on speech representations using LibriSpeech. Evaluating more diverse audio domains such as general audio (e.g., VGGSound) would strengthen the claim.
  3. Section 3.2 evaluates the models hypothesis only on one benchmark.
  4. In Section 3.1.1, the effect of changing the visual context in early layers appears somewhat different (not strictly CKA=1.0 but less than it) from that of changing only the text prompt, suggesting that some audio-visual interaction may already occur in these layers. In Section 3.2, it would therefore be informative to include pairwise encoding ablations, e.g., jointly encoding audio and vision while keeping text independent, rather than comparing only fully joint and fully independent encoding.
  5. The layer-wise representation analysis in Section 3.5 is limited to audio. Given the paper’s claim that both audio and visual encoders emerge, an analogous analysis of visual representations (e.g., from low-level visual features to object- and semantic-level representations) would provide more balanced evidence for the implicit vision encoder claim.
Questions for authors
  1. Section 3.2 evaluates the independent-vs.-joint encoding hypothesis only on OmniBench. Are there results on additional multimodal reasoning benchmarks that place stronger demands on audio-visual cross-modal interaction such as AVHBench (ICLR 2025)?
  2. Section 3.1.1 evaluates context invariance only on speech representations using LibriSpeech. Do the authors observe similar layer-wise context invariance for general audio, such as general audio (e.g., VGGSound) or music?
2 comments
AuthorsOpenReview ↗

Response to Reviewer z8r3

Thank you for your positive assessment, and for finding the study well motivated and the experimental design well structured. We respond to each point below.

Weakness #1 (Inkling-Small missing from MIEB). Thank you for catching this. This was an error in the text: Sec. 3.4 reports MIEB results for Gemma-4 12B only, and we will correct the sentence.

Weakness #2 / Question #2 (general audio and music). We chose LibriSpeech for the context-invariance analysis in Sec. 3.1.1 so that it would match the speech-based analyses that follow it (SUPERB in Sec. 3.3, and the phone- and word-level analysis in Sec. 3.5, which needs time-aligned transcripts). However, our cross-modal reasoning experiment already covers general audio. OmniBench includes speech, sound-event and music clips, and on these, independently encoding the audio up to layer 12 costs no accuracy (Figure 3). This shows that the implicit audio encoder does not depend on cross-modal context for non-speech audio either.

Weakness #3 / Question #1 (only one cross-modal benchmark). We chose OmniBench because it was built so that every question requires both the image and the audio to answer. We will look into re-running our analysis on AVHBench and report the results if we can finish them in time.

Weakness #4 (audio-visual interaction in early layers, and pairwise ablations). You are correct that not every curve in Figure 1 stays strictly at 1.0 through layer 12, and we will correct the paper accordingly. In Gemma-4 12B the dip around layer 8 appears in the conditions that change the text prompt (Transcription prompt and Prompt + photo). The condition that changes only the image (Real photo) stays close to 1.0 through layer 12. This suggests that the small early sensitivity comes mainly from the text context, not from audio-visual interaction.

Some early audio-visual interaction is architecturally possible. In Gemma-4 12B’s layout (text, then image, then audio), audio tokens can attend to image tokens from the first layer. However, Figure 3 shows that whatever interaction occurs is not functionally needed: fully independent encoding of all three modalities matches fully joint encoding up to layer 12. Since these two extremes already give the same accuracy up to layer 12, we would not expect a pairwise condition to differ.

Weakness #5 (layer-wise analysis of visual representations). We agree that a vision counterpart to Figure 5 would be informative. The layer-wise MIEB sweeps in Appendix A.1 (Figures 6-7) already show part of this picture. In Gemma-4 12B, average classification accuracy rises from about 0.19 at layer 1 to about 0.59 at layer 10, and average clustering NMI from about 0.21 to about 0.55, before both drop at layers 11-12. This mirrors the audio results in Figure 5, where phone and word identity peak around layers 8-9 and then fall. In both modalities, category-level information builds up through the implicit encoder layers and peaks just before cross-modal contextualization begins. We will move this discussion into the main text of Sec. 3.4. A finer-grained analysis of visual hierarchy (from low-level features to object and semantic representations) is a good direction, and we have added it to the future work in Sec. 4.

We thank you again for your constructive suggestions.

Thank you for your rebuttal. I will maintain my score.

Reviewer_ftYoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors investigate two encoder-free multimodal text-image-audio models, Gemma4-12B and Inkling-Small. The goal is to show that implicitly the first dozen layers of those models act as “encoders”. To that end, the authors show that modality representations barely change (up to a certain layer) when the surrounding context changes, as measured by CKA. They also experiment with encoding modalities in the first layers separately and then merging, and do some probing on layers. The results show that the modalities’ activations stay roughly unaffected by other inputs for the first few layers, and isolating them makes little difference.

Critical strengths

I find the approach and investigations genuinely interesting and worth studying. Whether encoder-free models end up relearning encoders internally is a timely question with practical implications. The different ways the models are probed with related/unrelated/no context make sense.

Critical weaknesses

For me, the main weakness is that the paper is not very clear about what an encoder is, or rather what properties we would expect from one, and concentrates mainly on evaluating whether image/audio features inside an encoder-free model are conditional or not. That doesn’t necessarily cover every aspect of what is usually understood of what an encoder should provide; one major aspect being learning “good” representations. For vision, the paper contains one set of experiments where they probe Gemma4-12B on MIEB(lite) image classification and clustering tasks, but I find that this setting is a bit small-scale and the paper could benefit from evaluating on standard vision/audio tasks like ImageNet cls, COCO/ADE segmentation, etc, both from frozen features, but also by finetuning. And then compare against baseline “encoders” like CLIP, DINOs, etc. There shouldn’t be an expectation that this way of transferring necessarily works as well, but to investigate whether implicitly an “encoder” is learned I find it to be important. Besides that, I think that there are some important baselines missing that could disambiguate the findings. For example, CKA staying around 1 for audio/img tokens only shows that these tokens don’t read from other modalities in every layer, but not he reverse; text tokens (for the actual output that is read out) may still attend and fuse information (doesn’t Fig 4 show that?). A simple baseline could be to do the same experiments within modalities, e.g. text-text with related or unrelated sentences, etc. If they also stay at CKA 1, this invariance is not specific to modalities. Also, what if the input is not the raw data, but from an encoder, e.g. from a more classical Llava-style model? Would those also delay fusion, and if yes, what would the paper’s conclusions be?

Questions for authors
  • How is CKA computed? Per token or pooled? Do the results change if this is changed?
  • In LLM activations there can sometimes be large outlier dimensions; is the analysis robust to that?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ftYo (part 1 of 2)

Thank you for your detailed review. We respond to each of your concerns below:

Weakness #1 (what is an encoder?). We agree the paper should state this explicitly, and we will add the following to Sec. 1. We define implicit encoder-like behavior by two principles:

  1. An encoder, first and foremost, learns abstract (aka "semantic") representations of the input that can support reasoning and other higher-level downstream functions. It is not obvious that this needs to happen in the first few layers of an encoder-free MLLM. Our claim (supported by empirical evidence) is that it does: in about 12 layers for Gemma and $n layers Inkling Small, as shown by linear probing.

  2. An encoder-free MLLM could be learning such representations (even within the first N layers) jointly from all modalities, which would make them fundamentally different from models that bolt on modality-specific encoders to a backbone LLM. Our claim is that this doesn't happen: the CKA analysis shows that MLLM representations after the first N layers are BOTH sufficiently abstract, a property "we should expect," and invariant to other-modality context, as shown by CKA etc.

Weakness #2 (larger-scale vision evaluation and fine-tuning). Our goal is not to show that these layers are competitive vision encoders, but that they compute an abstract, encoder-like representation. This is why we opt for evaluation of the frozen model layers on standard embedding benchmarks, namely SUPERB and MIEB. MIEB-lite already includes ImageNet-derived tasks (TinyImageNet and ImageNetDog15) and compares directly against SigLIP2 and DINOv2, as you suggest.

Fine-tuning would measure something different: what the layers can be adapted into, rather than what they already compute. A 2.7B-parameter, 12-layer Transformer could likely be fine-tuned into a strong classifier from many initializations, so the result would not tell us much about whether an encoder was learned implicitly. As you note, we also do not expect parity with dedicated encoders. SigLIP2 and DINOv2 are trained solely to produce image embeddings, while Gemma’s implicit encoder learns them as a by-product of next-token prediction. Even so, Gemma-4 12B Layer 10 matches or exceeds DINOv2-base on multiple tasks (Country211, PatchCamelyon and EuroSAT), and matches SigLIP2-base on TinyImageNet clustering (Table 2).

Weakness #3a (within-modality baseline). The concern, as we understand it, is that if early layers did very little contextualization of any kind, cross-modal invariance would follow automatically. Our results show this is not the case, and early layers do integrate context within a modality. In Figure 5, phone identity peaks at layer 8 and word identity at layer 9 in Gemma-4 12B. Recognizing a word requires aggregating information across many 40 ms input frames, so these layers must be attending heavily within the audio segment. Over the same layers, audio representations are unaffected by text and image context (Figure 1). This combination, strong within-modality integration with no cross-modal integration, is exactly what a modality-specific encoder does. The changed audio condition in Figure 1, whose CKA stays near 0 at every layer, also shows that the CKA metric is sensitive to changes in the audio content itself.

Weakness #3b (text tokens may still read from other modalities). We agree that CKA on image and audio tokens tests only one direction. However, our results already explicitly address the reverse direction in two ways:

• In our OmniBench inputs, the token layout follows the canonical Gemma 4 12B layout, which is: question text, then image, then audio. Under causal attention, the question tokens cannot attend to image or audio tokens at all, and image tokens cannot attend to audio. The only positions that can read from both modalities are those after the audio segment, where the answer is produced.

• The OmniBench experiment (Figure 3) blocks attention across modality segments in both directions up to the merge layer, including any text or answer tokens reading from image or audio. If that reverse-direction fusion mattered in early layers, accuracy would drop at early merge layers. It does not drop before about layer 12.

Weakness #3c (encoder-based, LLaVA-style models). Running our analysis on an encoder-full model would not directly test our claim, because it would not tell us whether encoder-free LLMs learn implicit encoders.

AuthorsOpenReview ↗

Response to Reviewer ftYo (part 2 of 2)

Question #1 (how is CKA computed?). For Figures 1 and 2, CKA is computed on pooled tokens, where the pooling is performed on a per-modality basis. We ran the analysis both ways, with pooled representations as well as token-level representations and observed that the CKA similarity curves are extremely similar regardless of which strategy is used. We will update the paper to more explicitly describe this.

Question #2 (outlier dimensions). Two observations suggest outliers are not driving our results:

• CKA centers each feature, so input-independent outliers (dimensions with large but constant values) are removed before similarity is computed. • The changed audio condition in Figure 1 stays near 0 at every layer. If a small set of outlier dimensions dominated the similarity, encoding a different utterance would also give high CKA. It does not, so the high CKA in the other conditions reflects the content of the representations.

We hope these clarifications address your soundness concerns, and we would be grateful if you would consider raising your score.

Thank you for the detailed rebuttal, which resolved several of my concerns.

Regarding my point about a text-text ablation, I don't think it was directly answered. My concern was not that early layers do little mixing overall, but whether the separation between segments is specific to modalities. From my understanding, the phone and word analysis in Fig 5 shows that early layers combine information across neighboring audio tokens. But that doesn’t show invariance like in the experiments across modalities; more just local mixing. In the response to Reviewer jUo4, you also acknowledge that context invariance might be a generic property of early LLM layers. Note that I don’t mean to say that this would contradict the full definition, but the paper describes these layers as modality-specific encoders that are rediscovered, and hypothesizes that fusion is delayed until an abstract representation has been computed. For those claims it matters whether the invariance is specific to modalities or not, as measured by CKA.

The controlled text-text experiment i meant could take a text segment, put either a related text (same document, previous sentence or tokens), an unrelated text from another document, or no text before it, and compute the same CKA on the fixed segment's tokens. An audio-audio version would work similarly, but I think a text-only version would be enough as a demonstration. If the fixed segment also stays at CKA around 1 until layer 12 or so, the results would suggest that early layers process segments locally in general, rather than in just the cross-modal way. If it changes early while image and audio tokens don't react to cross-modal context, that would in my opinion strengthen the paper's claims.

This is also why I asked about encoder-based models. If an LLM backbone that already receives "semantic" encoder tokens also delays fusion to a similar depth, the delay would not be explained by the need to first compute abstract features. I understand this comparison is not perfectly clean, but it would bring nuance to the claims and the author's chosen definition of what is an encoder.

Reviewer_jUo4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the early layer representation of Gemma-4 12B and Inkling-Small, two encoder-free early fusion MLLMs. The authors find that these early layers serve as an implicit vision/audio encoder. They first show that the representation of image/audio tokens is invariant to the context in the early layers. Then they evaluate the early-layer representation on standard encoder evaluation benchmarks and show they are comparable to specialized encoders on some tasks.

Critical strengths
  1. The paper is well motivated and nicely written, with clear research questions, hypotheses, and experiments.
  2. The experiments are generally well designed to prove the hypothesis.
  3. The finding that in Gemma4-12B the audio representation is invariant to image/text input in early layers is interesting and new. The performance as early layer as an encoder, evaluated on standard audio encoder benchmarks is high and well supports the hypothesis for Gemma-4 12B audio modality.
Critical weaknesses
  1. While the results are convincing for the audio modality, it is not the case for the vision modality, as evidenced by the results in Section 3.4. With the central claim being "early layers serving as an implicit encoder," the representation of the 10th layer is used for this experiment, with low performance on both vision tasks.
  2. It is not explained why Inkling's representation is invariant to context across all the layers except the last one, while results show a different trend in the activation patch experiment shown in Figure 3. The authors try to explain this with the boundary tokens, while in Figure 4, the similarity is low already from the very beginning for these boundary tokens; the implicit encoder within early layers hypothesis can not explained.
  3. It would be interesting to show whether the early-layer encoder is only valid for encoder-free MLLMs. For modular MLLMs, if you take the early layers features and evaluate on encoder benchmarks, what would the results look like?
Questions for authors
  1. What is the random-chance baseline for table 2
  2. What is the takeaway from the findings? How can it suggest the development of MLLMs?
3 comments
AuthorsOpenReview ↗

Response to reviewer jUo4 (Part 1)

Thank you for your feedback and suggestions. We reply to your questions one by one below:


Weakness #1 (vision results). The central claim of our paper (“early layers serving as an implicit encoder”) does not rest on those layers achieving state-of-the-art performance on embedding benchmarks. Our claim is that encoder-free multimodal LLMs rediscover implicit encoder-like behavior in their early layers. By this, we mean that within these layers, a low-level input (such as an image or an audio waveform) is processed according to two principles:

  1. The input is “privately” encoded. Changing the surrounding context does not change the encoded representation of the input. This mirrors how the audio (or image) encoder in a standard, encoder-full multimodal LLM encodes its input independently of the other modalities, up until the encoder outputs are concatenated and fed to the backbone LLM. The paper shows this for audio (Figure 1), for images (Figure 2), and for both modalities at once (Figure 3).

  2. The output is a high-level, abstract representation. It must contain enough semantic information for the input to be meaningfully contextualized with other modalities. The word “dog” cannot be semantically grounded to a single image patch, and the word “bark” cannot be grounded to a single 40 ms waveform window. The patches making up a photo of a dog must first be processed and grouped together before a semantic category like “dog” can emerge. Table 2 (images) and Table 1 and Figure 5 (audio) show that the features output by these layers are abstract enough to achieve nontrivial, and in some cases strong, performance on standard audio and image embedding benchmarks.

The first of these, private encoding, holds for vision just as it does for audio: image token representations are invariant to their textual context through the early layers (Figure 2), and encoding images independently up to layer 12 costs no accuracy on OmniBench (Figure 3).

On the second principle, the image features are weaker than the audio features overall, but they are far from uniformly weak. Gemma-4 12B Layer 10 matches DINOv2-base on Country211 (10.10 vs. 10.73) and PatchCamelyon (72.70 vs. 72.73), outperforms it on EuroSAT (83.20 vs. 81.66), and matches SigLIP2-base on TinyImageNet clustering (59.30 vs. 59.82). The gaps are concentrated on object-centric tasks such as OxfordPets and ImageNetDog15. We note that SigLIP2 and DINOv2 are trained solely to produce image embeddings, whereas Gemma’s implicit encoder learns them as a by-product of next-token prediction. We will update our manuscript to more clearly define what we mean by encoder-like behavior by discussing the two principles we outlined above.


Weakness #2 (Inkling and boundary tokens). You are correct that our text was inaccurate. The audio-boundary similarity in Figure 4 is already low (about 0.25) from layer 1. We will correct Sec. 3.2 accordingly. However, we think the corrected picture still supports the hypothesis:

• Boundary tokens are not modality content. In the independent condition they attend to padding instead of the neighboring segment, so their representations differ from the first layer. The implicit-encoder hypothesis concerns content tokens, and Figure 4 shows those stay near 1.0 up to the final layers.

• Figure 3 shows no accuracy loss on a tri-modal task when merging the representations from different, independently-encoded modalities before about layer 12, even though boundary tokens already differ. So whatever boundary tokens carry in early layers is not used downstream for this task. We hypothesize that after layer 12 the model starts routing cross-modal information through these tokens, which is why independent encoding past this point removes it and causes the task accuracy to drop.

AuthorsOpenReview ↗

Response to reviewer jUo4 (Part 2)

Weakness #3 (modular MLLMs). Our claim is that encoder-free multimodal LLMs rediscover implicit encoder-like behavior in their early layers. Within these layers, a low-level input (such as an image or an audio waveform) is processed according to the two principles we describe above in our response to Weakness #1 (Private encoding and highly abstracted representations).

If we understand your question correctly, evaluating the early LLM layers of an encoder-full model on encoder benchmarks would not directly test this claim, because it would not tell us whether encoder-free LLMs learn implicit encoders. We already know that the input embeddings of encoder-full LLMs do well on encoder benchmarks, since the models that do well on those benchmarks are the ones typically chosen as encoders. If benchmark results got worse after the early LLM layers, that would only tell us the LLM has started using those token positions for something different. If they got better, it would tell us the LLM has learned to make an already-powerful encoder more powerful. Both outcomes would be interesting, but neither bears on our claim.

A related question is whether principle 1 alone, context invariance, might simply be a generic property of early LLM layers. However, we define an "implicit encoder" as requiring both principle 1 and 2 to hold. In an encoder-full model, the backbone receives tokens that are already abstract, so invariance in its early layers says nothing about encoding. In an encoder-free model, the same layers must also transform raw pixel patches and waveform windows into abstract features, which Tables 1-2 and Figure 5 show they do. It is the combination of both principles that we interpret as an implicit encoder.


Question #1 (chance baseline). Chance accuracy for each classification task is 1/(number of classes): Country211 0.47%, DTD 2.13%, EuroSAT 10%, GTSRB 2.33%, OxfordPets 2.70%, PatchCamelyon 50%, RESISC45 2.22%, and SUN397 0.25%. For the two clustering tasks, the NMI of a random assignment is approximately 0. Averaged as in Table 2, chance is about 7.0, compared with 58.39 for Gemma-4 12B Layer 10, and every task is well above its own chance level. We will add these values to Table 2.


Question #2 (takeaways). We cover several takeaways in the conclusion, and will expand them in the revision. Our analysis shows that encoder-free LLMs are likely to “reinvent the wheel” each time they are trained, spending parameters and compute relearning representations that may be much the same as what a pre-trained, off-the-shelf encoder would have provided. Our distillation experiment (Sec. 3.6) supports this: the implicit audio encoder retains most of its SUPERB performance at about 7% of its original size.

A few actionable takeaways:

  1. Initializing the early layers of an encoder-free LLM with parameters from pre-trained encoders may speed up training and reduce the compute needed.

  2. Vision and audio encoders are typically not trained with a next-token prediction loss on text. Applying auxiliary losses like those used to train encoders (self-supervised objectives, or even supervised classification) specifically to these early layers could speed up training of encoder-free models.

  3. Encoder-free LLMs can be made more efficient at inference time. Because each modality can be encoded independently up to the middle layers without accuracy loss, those representations can be computed in parallel and cached for reuse across prompts. Computing cross-modal attention in these layers is wasted compute.

  4. Our results can also be read as critique of encoder-free architectures. If training an encoder-free model yields a model that rediscovers internal encoders anyway, the benefits of encoder-free over encoder-full architectures need to be more clearly established.

We hope these clarifications address your concerns, and we would be grateful if you would consider raising your score.

The authors address my concern well. I will increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.