- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to adapt a frozen text LLM to trimodal inputs, including speech and images. This is achieved by inserting trainable layers. The paper proposes MA-Conformer, which uses temporal convolution for speech tokens and spatial convolution for image tokens, and Expansion LayerDrop, which allows fewer added layers at inference. The paper reports better multimodal adaptation and less active-mode text degradation than several comparison methods.
Critical strengths
-
Keeping the original LLM frozen makes its text behavior exactly recoverable when the added layers are removed. This gives a clear way to separate base-model preservation from multimodal adaptation.
-
Matching the local operation to the structure of speech and image tokens is a natural design. The method also shares most expansion-layer parameters across modalities.
Critical weaknesses
- Limited technical contribution.
- The framework is a direct extension of Yano et al. (2026), which already inserts expansion layers into a frozen LLM for speech. The speech path of the MA-Conformer reuses the same cgMLP as the E-Branchformer layers used there.
- Every remaining component has close prior work that is either uncited or not compared against. For example, 2D convolution on the CLIP patch grid (Honeybee’s C-Abstractor).
- Overclaims & Limited evaluations.
- "Matching or surpassing Whisper-Medium" compares in-domain training (OWSM includes LibriSpeech) against Whisper’s zero-shot results. A 30M Conformer trained on LibriSpeech-960 already reaches 2.3/5.0.
- "A single trimodal model approaches unimodal models" is not a controlled comparison. The trimodal run uses about 6.4× the training tokens plus extra LibriSpeech data, yet still trails by 13–20% relative WER at 3B.
- The full FT and LoRA baselines appear under-tuned. Full FT seems to use the same lr of 1e-4 in speech, and LoRA uses r=356 applied to all tokens. Full FT’s vision score also collapses from 55.0 to 47.3 in the trimodal setting.
- Generality concerns.
- All experiments use one model family. Parameter-matched comparisons exist only at 1.7B, and Expansion LayerDrop is never tested in the trimodal setting.
- The trend across the two scales works against the main claims. For example, active-mode text degradation grows from −6.9 to −17.8 (1.7B -> 3B)
- Given the low training cost (~100 GPU-hours per speech run), adding another model family and multiple seeds is feasible.
- Two results are doubtful.
- The measured LayerDrop speedup (1.3× at 1.7B, 1.4× at 3B) exceeds the bound implied by layer count (1.2×), so the throughput measurement likely includes more than decoding.
- The claim of surpassing LLaVA-1.5 on ScienceQA may compare different subsets, since LLaVA-1.5 reports ScienceQA-IMG.
Questions for authors
- CLIP penultimate-layer features have already been mixed by more than twenty layers of global attention, so why would the expansion layers lack local spatial bias?
- Each speech frame consists of one SSL token and eight codec tokens. Are these flattened into the sequence or summed into one embedding per frame? If flattened, the 1D kernel (k=31) spans only about 3.4 frames and mixes codebook levels, so the “1D temporal sequence” motivation does not hold as stated. Please clarify the layout and the effective temporal receptive field.
8 comments
Weakness 2: Overclaims and limited evaluations
Comparison with Whisper
In the submitted version, Section 3.1.2 presented Whisper-Medium only “As a reference,” and Table 1 listed it in a separate row. None of our conclusions depends on this comparison. Our conclusions are based on the comparisons among adaptation methods trained on the same data. In the revised version, we have removed this comparison from the text of Section 3.1.2, and we have noted in the caption of Table 1 that OWSM v3.2 includes LibriSpeech while Whisper is evaluated in a zero-shot manner. ASR-only models such as the 30M Conformer (Conformer-M [10], 2.3/5.0 without an LM) mentioned in the review are also outside the scope of our comparison, because the goal of this paper is not state-of-the-art ASR but adapting an LLM to speech while keeping its text ability.
Trimodal vs. unimodal
As the review points out, this comparison is not matched in training tokens. In the revised version, App. C describes the training-token budget of each setting and states this point explicitly. The purpose of this comparison is to examine how close a single model that handles both modalities can get to two separately trained models. At 1.7B, the trimodal model uses 0.96B trainable parameters, which is only about 1.2 times that of a single unimodal model (0.81B) and much fewer than two unimodal models (1.62B) (Tables 1, 2, and 4). The remaining gap is small in absolute terms. The 13–20% relative gap at 3B corresponds to 0.4 and 0.6 WER points. Across both scales, the gap is at most 0.7 WER points and 1.7 Vision Avg. points (Section 3.3). In addition, the 6.4× figure counts the tokens of all modalities, and it does not mean 6.4× more speech data (App. C). Speech-only data accounts for one fifth of the mixture (Table 6), and the rest pairs text with images or with both images and speech. LibriSpeech is also included in OWSM v3.2 of the speech setting, so it is not a new data source.
Baseline tuning
As described in our response to Reviewer RXXq (Question 2), we used one shared configuration for each setting by design, and each configuration follows an established recipe. To examine whether the learning rate causes the gap in the speech setting, we additionally trained full fine-tuning with a ten times lower learning rate (1e-5, Table 9). As a result, lowering the learning rate reduces forgetting (Text Δ −27.5 vs. −32.5) but increases WER (7.2 vs. 5.6 on test-other). Therefore, at neither of the tested learning rates does full fine-tuning approach the MA-Conformer (Text Δ −6.9). In addition, as described below, the vision drop pointed out in the review is not specific to full fine-tuning.
The vision drop of full fine-tuning follows a clear pattern. From the image setting to the trimodal setting, the methods that adapt the base LLM itself (full fine-tuning and LoRA) lose much more vision accuracy than the depth up-scaling methods. At 1.7B, full fine-tuning and LoRA drop from 55.0 to 47.3 and from 49.8 to 47.1, while transformer expansion and the MA-Conformer stay almost unchanged (53.0 → 53.4 and 54.9 → 54.4). The same pattern holds at 3B (Tables 2 and 4). Both groups use the same learning rate in the trimodal setting. Under this recipe, the depth up-scaling methods lose much less vision performance than both methods that adapt the base LLM itself. Therefore, this drop does not indicate that only full fine-tuning is under-tuned.
Moreover, our main claim does not depend on this comparison. As shown in Tables 1, 2, and 4, depth up-scaling preserves the text ability better than full fine-tuning in all six setting–scale combinations, and this also holds for the additional full fine-tuning run with the lower learning rate (Table 9). In the image setting, every method already uses 1e-5. Even in this setting, full fine-tuning loses GSM8K accuracy (67.1 → 54.4), while the MA-Conformer keeps 66.9 at 3B (Table 5).
For LoRA, the rank r = 356 matches the number of trainable parameters of depth up-scaling at 1.7B (App. G). In the rank sweep, which was already reported in the appendix of the submitted version (Table 9), a lower rank forgets less but increases WER, and none of the ranks we tried matches the MA-Conformer on both ASR and text. The learning rate of 1e-4 is also within the standard range for LoRA (see our response to Reviewer RXXq, Question 2). Regarding the point that LoRA is applied to all tokens, the expansion layers also act on all tokens. The MHSA and FFN process text and modality tokens in the same way (Equations 3 and 5), and only the cgMLP is restricted to modality tokens.
Weakness 3: Generality
The review regards SmolLM2 and SmolLM3 as one model family. However, although they share the name, they differ in attention (multi-head vs. grouped-query), positional encoding (RoPE vs. RoPE with NoPE layers), tokenizer (49K vs. 128K vocabulary), and pre-training data. Our experiments cover these two base models, two scales, three settings, and up to eight methods, and the main comparisons hold in all of them. In addition, we conducted the parameter-matched comparisons at 1.7B in all three settings, including the trimodal setting where the parameter gap is the largest (Table 4). Expansion LayerDrop skips whole layers and does not depend on the modality, and we tested it on both convolution paths. This size range is also standard in studies of multimodal LLM design, such as TinyLLaVA [3] (1.1B to 2.7B) and SmolVLM [4] (256M to 2.2B). Moreover, our method only inserts layers into a standard decoder-only transformer, and nothing in it is specific to SmolLM. Regarding the uncertainty over the test sets, our response to Reviewer RXXq (Uncertainty of small gaps) reports bootstrap confidence intervals. The WER reduction is significant on test-other at both scales and on test-clean at 3B. Note that about 100 GPU-hours is the cost of a single speech run at 1.7B, and the training-token budget of a single trimodal run is 6.4 times larger (App. C).
As stated in the Limitations of the submitted version, Expansion LayerDrop was not evaluated in the trimodal setting. Therefore, we additionally evaluated it during the rebuttal period. We continued training the trimodal MA-Conformer checkpoint on SmolLM2-1.7B with Ordered Drop in a single run. The training data is a mixture of the data used in Section 3.4.1 (3.8k hours of speech and one additional epoch of the image data), and the optimizer settings, including the peak learning rate of 1e-4, follow the trimodal training (App. C). The table below shows the results at representative values of k. The throughput is measured with image inputs under the same conditions as Figure 2.
| k | WER clean ↓ | Vision Avg. ↑ | Throughput (tok/s) ↑ |
|---|---|---|---|
| 6 (= K) | 2.5 | 55.6 | 88 |
| 3 | 3.6 | 55.2 | 103 |
| 1 | 8.1 | 51.8 | 115 |
As shown in the table, training with Ordered Drop does not degrade the performance at k = K compared with the model before LayerDrop training (WER 2.5 and Vision Avg. 54.4 in Table 4). In addition, as k decreases, both WER and Vision Avg. degrade gradually, and the model does not collapse even at k = 1. These results are consistent with those in the unimodal settings (Section 3.4 and Figure 2). Therefore, the main conclusion of Section 3.4, that a single trained model can serve multiple deployment scenarios by choosing k, also holds in the trimodal setting with shared expansion layers. We have added this result to App. L (Table 14), referred to it in Section 3.4, and updated the Limitations in the revised version.
In addition, the trend across scales does not contradict our claims. Our claim is relative to full fine-tuning, and full fine-tuning also degrades more at 3B in every setting. The margin over full fine-tuning becomes larger in the image and trimodal settings and remains above 20 points in the speech setting. Moreover, skipping the expansion layers and removing the added vocabulary entries recovers the base model exactly at both scales.
We thank you for your review. We have uploaded a revised version of the paper, and all section, table, and appendix numbers below refer to the revised version. We would like to address your concerns.
Weakness 1: Limited technical contribution
We respectfully disagree that the technical contribution is limited. As the review points out, our speech path reuses the cgMLP of E-Branchformer used in Yano et al. [1] (Section 2.2 and App. D). However, Yano et al. [1] study only speech, and we count neither the speech setting nor the cgMLP itself as our contribution. We would like to clarify that the main contribution of this paper is the design principle of the inserted layers. The review notes that matching the local operation to the structure of speech and image tokens is “a natural design.” However, whether this design actually matters inside a frozen LLM had not been examined. To the best of our knowledge, this paper is the first work that shows it. Specifically, the 2D kernel outperforms the 1D kernel on image tokens with the same parameter budget (0.81B, Table 3). In addition, the new controls in Table 3 (see also our response to Reviewer RXXq, Question 1) show that this improvement cannot be explained only by added capacity, gating, or non-causal processing.
For the other components, we have already cited the closely related works in Section 2.4 and Section 4, and our response to Reviewer RXXq (Weakness 1) describes the differences from each of them. We thank the reviewer for pointing out Honeybee’s C-Abstractor [2], and we have added it to Section 2.2 and Section 4 in the revised version. However, C-Abstractor addresses a different problem from ours. C-Abstractor processes the CLIP features only once in the projector before the LLM, and Honeybee then fine-tunes the LLM. In contrast, we keep all tokens, freeze the LLM, and switch the local operation depending on the modality inside every expansion layer. Moreover, Honeybee reports that a linear projector already preserves local context. Our image setting uses such a linear projector (App. B). Even in this setting, the 2D kernel inside the LLM still outperforms the 1D kernel. Therefore, the improvement is obtained by the local processing inside the LLM, without changing the projector.