- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper anlayze whether the early layers of encoder-free multimodal LLMs function similary as the dedicated audio and visual encoders. The extensive results shows the evidence of these early layers act as implicit encoders.
Critical strengths
Well-motivated study of a timely and relatively underexplored problem in encoder-free multimodal LLMs. The experimental design is well structured and comprehensive.
Critical weaknesses
- Section 3.4 states that the visual representations of both Gemma-4 12B and Inkling-Small are evaluated as vision encoders, yet the reported MIEB results and the appendix only include Gemma-4 12B.
- Section 3.1.1 evaluates context invariance only on speech representations using LibriSpeech. Evaluating more diverse audio domains such as general audio (e.g., VGGSound) would strengthen the claim.
- Section 3.2 evaluates the models hypothesis only on one benchmark.
- In Section 3.1.1, the effect of changing the visual context in early layers appears somewhat different (not strictly CKA=1.0 but less than it) from that of changing only the text prompt, suggesting that some audio-visual interaction may already occur in these layers. In Section 3.2, it would therefore be informative to include pairwise encoding ablations, e.g., jointly encoding audio and vision while keeping text independent, rather than comparing only fully joint and fully independent encoding.
- The layer-wise representation analysis in Section 3.5 is limited to audio. Given the paper’s claim that both audio and visual encoders emerge, an analogous analysis of visual representations (e.g., from low-level visual features to object- and semantic-level representations) would provide more balanced evidence for the implicit vision encoder claim.
Questions for authors
- Section 3.2 evaluates the independent-vs.-joint encoding hypothesis only on OmniBench. Are there results on additional multimodal reasoning benchmarks that place stronger demands on audio-visual cross-modal interaction such as AVHBench (ICLR 2025)?
- Section 3.1.1 evaluates context invariance only on speech representations using LibriSpeech. Do the authors observe similar layer-wise context invariance for general audio, such as general audio (e.g., VGGSound) or music?
2 comments
Thank you for your rebuttal. I will maintain my score.
Response to Reviewer z8r3
Thank you for your positive assessment, and for finding the study well motivated and the experimental design well structured. We respond to each point below.
Weakness #1 (Inkling-Small missing from MIEB). Thank you for catching this. This was an error in the text: Sec. 3.4 reports MIEB results for Gemma-4 12B only, and we will correct the sentence.
Weakness #2 / Question #2 (general audio and music). We chose LibriSpeech for the context-invariance analysis in Sec. 3.1.1 so that it would match the speech-based analyses that follow it (SUPERB in Sec. 3.3, and the phone- and word-level analysis in Sec. 3.5, which needs time-aligned transcripts). However, our cross-modal reasoning experiment already covers general audio. OmniBench includes speech, sound-event and music clips, and on these, independently encoding the audio up to layer 12 costs no accuracy (Figure 3). This shows that the implicit audio encoder does not depend on cross-modal context for non-speech audio either.
Weakness #3 / Question #1 (only one cross-modal benchmark). We chose OmniBench because it was built so that every question requires both the image and the audio to answer. We will look into re-running our analysis on AVHBench and report the results if we can finish them in time.
Weakness #4 (audio-visual interaction in early layers, and pairwise ablations). You are correct that not every curve in Figure 1 stays strictly at 1.0 through layer 12, and we will correct the paper accordingly. In Gemma-4 12B the dip around layer 8 appears in the conditions that change the text prompt (Transcription prompt and Prompt + photo). The condition that changes only the image (Real photo) stays close to 1.0 through layer 12. This suggests that the small early sensitivity comes mainly from the text context, not from audio-visual interaction.
Some early audio-visual interaction is architecturally possible. In Gemma-4 12B’s layout (text, then image, then audio), audio tokens can attend to image tokens from the first layer. However, Figure 3 shows that whatever interaction occurs is not functionally needed: fully independent encoding of all three modalities matches fully joint encoding up to layer 12. Since these two extremes already give the same accuracy up to layer 12, we would not expect a pairwise condition to differ.
Weakness #5 (layer-wise analysis of visual representations). We agree that a vision counterpart to Figure 5 would be informative. The layer-wise MIEB sweeps in Appendix A.1 (Figures 6-7) already show part of this picture. In Gemma-4 12B, average classification accuracy rises from about 0.19 at layer 1 to about 0.59 at layer 10, and average clustering NMI from about 0.21 to about 0.55, before both drop at layers 11-12. This mirrors the audio results in Figure 5, where phone and word identity peak around layers 8-9 and then fall. In both modalities, category-level information builds up through the implicit encoder layers and peaks just before cross-modal contextualization begins. We will move this discussion into the main text of Sec. 3.4. A finer-grained analysis of visual hierarchy (from low-level features to object and semantic representations) is a good direction, and we have added it to the future work in Sec. 4.
We thank you again for your constructive suggestions.