acceptodds
Under review as a conference paper at ICLR 2027

Modality-Adaptive Depth Up-Scaling for Trimodal LLM Adaptation

Abstract

Adapting pre-trained text large language models (LLMs) to new modalities via continual pre-training often degrades their text capabilities. Depth up-scaling, which inserts new layers into a pre-trained LLM, has gained attention for adding capacity while preserving the original capabilities, and has been shown to be effective for adapting text LLMs to speech. We present modality-adaptive depth up-scaling, which extends this approach to a trimodal (text, speech, and image) setting, in which a single text LLM is adapted to both speech and image through shared expansion layers. Since speech and image tokens form a 1D temporal sequence and a 2D spatial grid, respectively, we introduce modality-adaptive (MA) operations, namely 1D convolution for speech tokens and 2D convolution for image tokens, and build each expansion layer as an MA-Conformer layer that switches between them by token type. We also introduce Expansion LayerDrop, which lets a single trained model trade performance for speed by activating fewer expansion layers at inference. On speech-to-text, image-to-text, and jointly trained trimodal adaptation with 1.7B and 3B base LLMs, MA-Conformer expansion layers outperform standard transformer expansion layers and cause less text degradation than full fine-tuning, and a single trimodal model approaches the performance of separately trained unimodal models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal llm, speech, vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bzLKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to adapt a frozen text LLM to trimodal inputs, including speech and images. This is achieved by inserting trainable layers. The paper proposes MA-Conformer, which uses temporal convolution for speech tokens and spatial convolution for image tokens, and Expansion LayerDrop, which allows fewer added layers at inference. The paper reports better multimodal adaptation and less active-mode text degradation than several comparison methods.

Critical strengths
  1. Keeping the original LLM frozen makes its text behavior exactly recoverable when the added layers are removed. This gives a clear way to separate base-model preservation from multimodal adaptation.

  2. Matching the local operation to the structure of speech and image tokens is a natural design. The method also shares most expansion-layer parameters across modalities.

Critical weaknesses
  1. Limited technical contribution.
    • The framework is a direct extension of Yano et al. (2026), which already inserts expansion layers into a frozen LLM for speech. The speech path of the MA-Conformer reuses the same cgMLP as the E-Branchformer layers used there.
    • Every remaining component has close prior work that is either uncited or not compared against. For example, 2D convolution on the CLIP patch grid (Honeybee’s C-Abstractor).
  2. Overclaims & Limited evaluations.
    • "Matching or surpassing Whisper-Medium" compares in-domain training (OWSM includes LibriSpeech) against Whisper’s zero-shot results. A 30M Conformer trained on LibriSpeech-960 already reaches 2.3/5.0.
    • "A single trimodal model approaches unimodal models" is not a controlled comparison. The trimodal run uses about 6.4× the training tokens plus extra LibriSpeech data, yet still trails by 13–20% relative WER at 3B.
    • The full FT and LoRA baselines appear under-tuned. Full FT seems to use the same lr of 1e-4 in speech, and LoRA uses r=356 applied to all tokens. Full FT’s vision score also collapses from 55.0 to 47.3 in the trimodal setting.
  3. Generality concerns.
    • All experiments use one model family. Parameter-matched comparisons exist only at 1.7B, and Expansion LayerDrop is never tested in the trimodal setting.
    • The trend across the two scales works against the main claims. For example, active-mode text degradation grows from −6.9 to −17.8 (1.7B -> 3B)
    • Given the low training cost (~100 GPU-hours per speech run), adding another model family and multiple seeds is feasible.
  4. Two results are doubtful.
    • The measured LayerDrop speedup (1.3× at 1.7B, 1.4× at 3B) exceeds the bound implied by layer count (1.2×), so the throughput measurement likely includes more than decoding.
    • The claim of surpassing LLaVA-1.5 on ScienceQA may compare different subsets, since LLaVA-1.5 reports ScienceQA-IMG.
Questions for authors
  1. CLIP penultimate-layer features have already been mixed by more than twenty layers of global attention, so why would the expansion layers lack local spatial bias?
  2. Each speech frame consists of one SSL token and eight codec tokens. Are these flattened into the sequence or summed into one embedding per frame? If flattened, the 1D kernel (k=31) spans only about 3.4 frames and mixes codebook levels, so the “1D temporal sequence” motivation does not hold as stated. Please clarify the layout and the effective temporal receptive field.
8 comments

We thank you for your review. We have uploaded a revised version of the paper, and all section, table, and appendix numbers below refer to the revised version. We would like to address your concerns.

Weakness 1: Limited technical contribution

We respectfully disagree that the technical contribution is limited. As the review points out, our speech path reuses the cgMLP of E-Branchformer used in Yano et al. [1] (Section 2.2 and App. D). However, Yano et al. [1] study only speech, and we count neither the speech setting nor the cgMLP itself as our contribution. We would like to clarify that the main contribution of this paper is the design principle of the inserted layers. The review notes that matching the local operation to the structure of speech and image tokens is “a natural design.” However, whether this design actually matters inside a frozen LLM had not been examined. To the best of our knowledge, this paper is the first work that shows it. Specifically, the 2D kernel outperforms the 1D kernel on image tokens with the same parameter budget (0.81B, Table 3). In addition, the new controls in Table 3 (see also our response to Reviewer RXXq, Question 1) show that this improvement cannot be explained only by added capacity, gating, or non-causal processing.

For the other components, we have already cited the closely related works in Section 2.4 and Section 4, and our response to Reviewer RXXq (Weakness 1) describes the differences from each of them. We thank the reviewer for pointing out Honeybee’s C-Abstractor [2], and we have added it to Section 2.2 and Section 4 in the revised version. However, C-Abstractor addresses a different problem from ours. C-Abstractor processes the CLIP features only once in the projector before the LLM, and Honeybee then fine-tunes the LLM. In contrast, we keep all tokens, freeze the LLM, and switch the local operation depending on the modality inside every expansion layer. Moreover, Honeybee reports that a linear projector already preserves local context. Our image setting uses such a linear projector (App. B). Even in this setting, the 2D kernel inside the LLM still outperforms the 1D kernel. Therefore, the improvement is obtained by the local processing inside the LLM, without changing the projector.

Weakness 2: Overclaims and limited evaluations

Comparison with Whisper

In the submitted version, Section 3.1.2 presented Whisper-Medium only “As a reference,” and Table 1 listed it in a separate row. None of our conclusions depends on this comparison. Our conclusions are based on the comparisons among adaptation methods trained on the same data. In the revised version, we have removed this comparison from the text of Section 3.1.2, and we have noted in the caption of Table 1 that OWSM v3.2 includes LibriSpeech while Whisper is evaluated in a zero-shot manner. ASR-only models such as the 30M Conformer (Conformer-M [10], 2.3/5.0 without an LM) mentioned in the review are also outside the scope of our comparison, because the goal of this paper is not state-of-the-art ASR but adapting an LLM to speech while keeping its text ability.

Trimodal vs. unimodal

As the review points out, this comparison is not matched in training tokens. In the revised version, App. C describes the training-token budget of each setting and states this point explicitly. The purpose of this comparison is to examine how close a single model that handles both modalities can get to two separately trained models. At 1.7B, the trimodal model uses 0.96B trainable parameters, which is only about 1.2 times that of a single unimodal model (0.81B) and much fewer than two unimodal models (1.62B) (Tables 1, 2, and 4). The remaining gap is small in absolute terms. The 13–20% relative gap at 3B corresponds to 0.4 and 0.6 WER points. Across both scales, the gap is at most 0.7 WER points and 1.7 Vision Avg. points (Section 3.3). In addition, the 6.4× figure counts the tokens of all modalities, and it does not mean 6.4× more speech data (App. C). Speech-only data accounts for one fifth of the mixture (Table 6), and the rest pairs text with images or with both images and speech. LibriSpeech is also included in OWSM v3.2 of the speech setting, so it is not a new data source.

Baseline tuning

As described in our response to Reviewer RXXq (Question 2), we used one shared configuration for each setting by design, and each configuration follows an established recipe. To examine whether the learning rate causes the gap in the speech setting, we additionally trained full fine-tuning with a ten times lower learning rate (1e-5, Table 9). As a result, lowering the learning rate reduces forgetting (Text Δ −27.5 vs. −32.5) but increases WER (7.2 vs. 5.6 on test-other). Therefore, at neither of the tested learning rates does full fine-tuning approach the MA-Conformer (Text Δ −6.9). In addition, as described below, the vision drop pointed out in the review is not specific to full fine-tuning.

The vision drop of full fine-tuning follows a clear pattern. From the image setting to the trimodal setting, the methods that adapt the base LLM itself (full fine-tuning and LoRA) lose much more vision accuracy than the depth up-scaling methods. At 1.7B, full fine-tuning and LoRA drop from 55.0 to 47.3 and from 49.8 to 47.1, while transformer expansion and the MA-Conformer stay almost unchanged (53.0 → 53.4 and 54.9 → 54.4). The same pattern holds at 3B (Tables 2 and 4). Both groups use the same learning rate in the trimodal setting. Under this recipe, the depth up-scaling methods lose much less vision performance than both methods that adapt the base LLM itself. Therefore, this drop does not indicate that only full fine-tuning is under-tuned.

Moreover, our main claim does not depend on this comparison. As shown in Tables 1, 2, and 4, depth up-scaling preserves the text ability better than full fine-tuning in all six setting–scale combinations, and this also holds for the additional full fine-tuning run with the lower learning rate (Table 9). In the image setting, every method already uses 1e-5. Even in this setting, full fine-tuning loses GSM8K accuracy (67.1 → 54.4), while the MA-Conformer keeps 66.9 at 3B (Table 5).

For LoRA, the rank r = 356 matches the number of trainable parameters of depth up-scaling at 1.7B (App. G). In the rank sweep, which was already reported in the appendix of the submitted version (Table 9), a lower rank forgets less but increases WER, and none of the ranks we tried matches the MA-Conformer on both ASR and text. The learning rate of 1e-4 is also within the standard range for LoRA (see our response to Reviewer RXXq, Question 2). Regarding the point that LoRA is applied to all tokens, the expansion layers also act on all tokens. The MHSA and FFN process text and modality tokens in the same way (Equations 3 and 5), and only the cgMLP is restricted to modality tokens.

Weakness 3: Generality

The review regards SmolLM2 and SmolLM3 as one model family. However, although they share the name, they differ in attention (multi-head vs. grouped-query), positional encoding (RoPE vs. RoPE with NoPE layers), tokenizer (49K vs. 128K vocabulary), and pre-training data. Our experiments cover these two base models, two scales, three settings, and up to eight methods, and the main comparisons hold in all of them. In addition, we conducted the parameter-matched comparisons at 1.7B in all three settings, including the trimodal setting where the parameter gap is the largest (Table 4). Expansion LayerDrop skips whole layers and does not depend on the modality, and we tested it on both convolution paths. This size range is also standard in studies of multimodal LLM design, such as TinyLLaVA [3] (1.1B to 2.7B) and SmolVLM [4] (256M to 2.2B). Moreover, our method only inserts layers into a standard decoder-only transformer, and nothing in it is specific to SmolLM. Regarding the uncertainty over the test sets, our response to Reviewer RXXq (Uncertainty of small gaps) reports bootstrap confidence intervals. The WER reduction is significant on test-other at both scales and on test-clean at 3B. Note that about 100 GPU-hours is the cost of a single speech run at 1.7B, and the training-token budget of a single trimodal run is 6.4 times larger (App. C).

As stated in the Limitations of the submitted version, Expansion LayerDrop was not evaluated in the trimodal setting. Therefore, we additionally evaluated it during the rebuttal period. We continued training the trimodal MA-Conformer checkpoint on SmolLM2-1.7B with Ordered Drop in a single run. The training data is a mixture of the data used in Section 3.4.1 (3.8k hours of speech and one additional epoch of the image data), and the optimizer settings, including the peak learning rate of 1e-4, follow the trimodal training (App. C). The table below shows the results at representative values of k. The throughput is measured with image inputs under the same conditions as Figure 2.

kWER clean ↓Vision Avg. ↑Throughput (tok/s) ↑
6 (= K)2.555.688
33.655.2103
18.151.8115

As shown in the table, training with Ordered Drop does not degrade the performance at k = K compared with the model before LayerDrop training (WER 2.5 and Vision Avg. 54.4 in Table 4). In addition, as k decreases, both WER and Vision Avg. degrade gradually, and the model does not collapse even at k = 1. These results are consistent with those in the unimodal settings (Section 3.4 and Figure 2). Therefore, the main conclusion of Section 3.4, that a single trained model can serve multiple deployment scenarios by choosing k, also holds in the trimodal setting with shared expansion layers. We have added this result to App. L (Table 14), referred to it in Section 3.4, and updated the Limitations in the revised version.

In addition, the trend across scales does not contradict our claims. Our claim is relative to full fine-tuning, and full fine-tuning also degrades more at 3B in every setting. The margin over full fine-tuning becomes larger in the image and trimodal settings and remains above 20 points in the speech setting. Moreover, skipping the expansion layers and removing the added vocabulary entries recovers the base model exactly at both scales.

Reviewer_RXXqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies depth up-scaling—adding trainable warm-initialized layers to a frozen model—for multimodal adaptation of LLMs. The authors extend previous work by Yano et al. (2026) that applied depth-upscaling for adapting a text LLM for automatic speech recognition (ASR) to image adaptation and joint image and speech adaptation. Beyond the new application domains, the authors change the architecture of the adaptation layers: while Yano et al. used E-Branchformer blocks, they propose the use of modality-adaptive conformer (MA-Conformer) blocks, which combine shared attention and FFNs with modality-specific convolutional gated layers. MA-Conformer uses 2D convolutions for images and 1D convolutions for speech. The authors also introduce adapter-layer dropout training to support variable depth at inference time. Experiments on SmolLM 1.7B and 3B report improvements over other adaptation layer architectures, including the E-Branchformer and conventional transformer layers, both in terms of target-modality performance, and in better preserving text-domain performance relative to full fine-tuning. The joint image and speech adapted model approaches the performance of the models adapter for just a single target modality.

Critical strengths

The paper presents reasonable and consistent evidence that its adaptation layer architecture improves upon regular transformer layers. Importantly, the evidence holds across modalities and for the trimodal setup. The controls largely support this, with the parameter-matched comparisons being particularly informative and ruling out trivial explanations, and the baselines against which they compare make a compelling case over other standard adaptation architectures, including Flamingo-style adapters and LoRA adapters.

The paper is also well motivated. Sharing most adaptation parameters across modalities could reduce the memory footprint of multimodal models, and variable-depth at inference time addresses the inference compute overhead introduced by the adapter layers, one of the most important issues of depth-upcycling. The results demonstrate working approaches towards both goals.

Critical weaknesses

Methodologically, the paper is largely incremental. Yano et al. already proposed frozen depth expansion with speech-specific convolutional processing, and IAA (Wang et al., 2025, not cited) already demonstrated visual adaptation through depth-upcycling. Modality-specific adaptation with shared global attention for cross-modal interaction also has clear precedents in works like Mixture-of-Transformers and, for PEFT methods, with MM-LoRA (Sun et al. 2024). The present paper is mostly a combination of these ideas, with the most meaningful contribution being showing the feasibility of trimodal depth-upcycling, together with adapter dropout training, which also has literature precedents. None of the precedents establish the same complete method presented in the paper. However, it is not clear to me that their combination yields a substantial new insight, and the performance gains require further validation.

The experiments do not sufficiently establish the cause or practical significance of the improvements. Most of the performance improvements over standard transformer adapters are generally modest and lack uncertainty estimates. The parameter-matched comparisons at the 1.7B scale are useful, but do not cleanly isolate the effect of modality-specific parameters, gating layers, convolutional layers, and non-causal mixing. The proposed layers add several of those variables at once. At 3B this is further emphasized, as there are no parameter-matched comparisons. Hyperparameter selection is also under-documented and likely suboptimal for the baselines, with LoRA explicitly stated as inheriting the depth-upcycling optimization settings. Therefore, its weaker adaptation-retention trade-off could be due to poor hyperparameters.

Other less critical but still meaningful weaknesses:

The practical baseline comparison is incomplete in my opinion. Full fine-tuning and LoRA are evaluated without the common practice of forgetting mitigation mechanisms. While this is a legitimate control, comparisons with text replay or KL constraints towards the original model’s predictions would help establish whether the architecture offers a practical advantage over alternative more standard adaptation methods.

The adapter dropout evaluation omits the original checkpoint, continuation training without dropout, and potentially separately trained fixed-depth models as baselines. It therefore does not establish the cost of supporting multiple inference budgets.

Finally, the substantial GSM8K degradation in the speech setting makes the missing generative-text comparisons for trimodal models and transformer adapters more relevant.

Questions for authors

Most important:

  • Can you provide parameter.matched ablations that isolate the contributions of the gating layers vs. the convolutions—e.g. replacing the convolution with an element-wise operation— and assess whether allowing for non-causal processing—e.g. through bidirectional attention within target-modality inputs—accounts for parts of the gain?
  • Please describe in more detail the hyperparameter selection strategies, and if the methods were sufficiently tuned for a fair comparison.
  • Please discuss the relevance of the contributions considering that all of the proposed methods have clear precedents in the literature.

Important, but less priority:

  • How does the method compare regarding the adaptation-retention trade-off with more standard fine-tuning recipes that include strategies for mitigation forgetting of text capabilities?
  • Can you experimentally assess the performance cost of variable depth training?
  • Can you report GSM8K results for setups other than the speech-adapted model?

Edit: I have updated my rating to Weak Accept.

8 comments

We thank you for your review. Following your suggestions, we conducted four additional experiments, namely component ablations (Question 1), full fine-tuning with a lower learning rate (Question 2), a continuation control without dropping (Weakness 4), and GSM8K evaluation of the trimodal and transformer-expansion models (Weakness 5). We have uploaded a revised version that includes these results, and all section, table, and appendix numbers below refer to the revised version. We would like to address your concerns below.

Weakness 1 & Question 3: Novelty

As described in Section 4, we have already cited the closely related works such as Mixture-of-Transformers, mPLUG-Owl2, CogVLM, SOLAR, and LLaMA Pro. In the revised version, we have also added IAA and MM-LoRA to Section 4.

However, we would like to clarify that the main point of this paper is not only the feasibility of trimodal depth up-scaling. As stated in the Introduction, the central proposal of this paper is “matching the local operation of the expansion layers to the structure of each modality.” Speech and images have different structures (a 1D sequence for speech and a 2D grid for images), and we claim that the inserted layers should process each modality according to this structure. We believe that this point is our answer to the question of whether the combination of these ideas “yields a substantial new insight.”

To the best of our knowledge, this paper is the first work that shows the following three findings. These findings are obtained through controlled comparisons in three settings and at two scales, and we use parameter-matched controls in all three settings at 1.7B.

  1. The local operation of the inserted layers should be designed according to each modality (Tables 2 and 3, and Question 1 below).
  2. A single shared set of inserted layers can handle both speech and images. It requires 0.96B trainable parameters, while two unimodal models require 1.62B in total (Table 4).
  3. Among the compared drop schedules, only Ordered Drop, which trains on the same nested subsets used at inference, does not collapse when the depth is reduced (Section 3.4).

The differences from the related works are as follows.

  • Yano et al. [1] focus only on speech, and they do not examine whether the local operation should be changed depending on the modality. Note that our speech setting follows theirs only for comparison.
  • IAA [2] inserts copies of transformer layers into a frozen LLM, but only for vision. Its inserted layers have the same basic form as our transformer expansion baseline, and the parameter-matched MA-Conformer outperforms this baseline in all three settings. In addition, IAA focuses on how the inserted layers are connected to the frozen path. In contrast, we focus on what the inserted layers should compute for each modality.
  • Mixture-of-Transformers [3] and MM-LoRA [4] separate linear projections, FFNs, or LoRA modules by modality. However, neither of them introduces an explicit local operation over neighboring tokens, such as convolution. Moreover, in Table 2, the attention-side counterpart (modality-specific key/value projections as in mPLUG-Owl2) performs worse than plain transformer expansion.
  • Layer dropping methods (LayerDrop, ordered dropout, and LayerSkip, cited in Section 2.4) drop layers over the whole network or in the width direction. On the other hand, Expansion LayerDrop drops only the inserted layers. Therefore, the original LLM can be exactly recovered.

From the above, we believe that our paper provides new knowledge that cannot be obtained from these previous works, and that it gives concrete design principles for adapting frozen LLMs to new modalities.

Weakness 2: Source of the gains

Below, Question 1 explains where the gain comes from, and Question 2 explains how we set the hyperparameters. After that, we discuss the uncertainty of gaps.

Question 1: Component ablations

One of the factors behind the advantage of MA-Conformer was already examined in the submitted version. As shown in Table 3, in the comparison between the 1D and 2D convolutions at 0.81B, all the other factors are identical and only the kernel shape on the image grid is different. The results show that the 2D kernel outperforms the 1D kernel. This suggests that the 2D locality on the image grid, which is inherent to 2D convolution, is important. For the remaining factors, we performed additional experiments, because we believe that the reviewer's suggestions make the paper more meaningful. First, as suggested by the reviewer, we replaced the convolution with an element-wise operation (a 1×1 kernel). In this setting, the gating mechanism is kept, but spatial mixing is not performed. Next, we conducted an experiment that introduces only bidirectional attention within the image tokens. In this setting, only the MHSA of the expansion layers uses a bidirectional mask within the image tokens, while the frozen original layers keep the causal mask. Therefore, the model processes the image tokens non-causally in the expansion layers but does not use convolution. All the other settings follow the image setting in Table 2. Note that the output projection of the cgMLP is initialized to zero (App. A). Therefore, all the convolutional variants in this table start from the same function as transformer expansion, and the differences between them come from what each branch learns. The following table, which we have added as Table 3 in the revised version, shows the image-to-text results on SmolLM2-1.7B with the INTERLEAVED placement.

Expansion layerGatingLocal mixingNon-causalTrainableVal. Loss ↓Vision Avg. ↑Text Δ
Transformer———0.66B2.22553.0−0.1
Transformer, bidirectional within image tokens——✓0.66B2.22752.5−0.2
Conformer, 1×1 kernel (new)✓——0.81B2.23152.0−0.1
Conformer, 1D kernel✓1D✓0.81B2.17154.0−0.2
MA-Conformer, 3×3 kernel✓2D✓0.81B2.15754.9−0.1

As shown in the table, neither the 1×1 kernel nor bidirectional attention within the image tokens outperforms transformer expansion. Both of them are at the same level as or slightly below transformer expansion in terms of validation loss and Vision Avg. In particular, the 1×1 kernel adds 0.15B parameters and the gating mechanism, but it does not bring any improvement. On the other hand, the 1D and 2D kernels reduce the validation loss to 2.171 and 2.157, respectively, with the same parameter budget (0.81B) as the 1×1 kernel. From these results, we consider that the improvement cannot be explained only by added capacity, gating, or non-causal processing, and that it comes from local mixing that matches the token structure. Note that we conducted the parameter-matched comparisons at 1.7B in all three settings, including the trimodal setting where the parameter gap is the largest (Table 4). Therefore, we believe that the results at 1.7B are sufficient to support our claim, and we expect that the same conclusion also holds at 3B.

Question 2: Hyperparameter tuning

We used one shared configuration for each setting by design. Our research question is how the trained part of the model should be designed. Therefore, every method uses the same optimizer, schedule, and number of steps, and our method does not receive any special treatment. Each configuration follows an established recipe, not a recipe tuned for our method. For speech and trimodal, we use a peak learning rate of 1e-4 with 10k warmup steps, following Yano et al. [1]. For image, we use 1e-5 with 10% warmup. This is close to 2e-5, which LLaVA-1.5 uses when it updates the full LLM [5]. Therefore, in the image setting, full fine-tuning is trained with a learning rate close to its own standard recipe. It is also known that even this recipe degrades the text ability. IAA reports drops on MMLU and C-Eval after LLaVA-1.5 training with both Qwen2 and Llama3 [2]. A per-method search on top of these recipes was not possible within our compute budget, because a single trimodal run alone has a budget of up to 32B training tokens (App. C). We have described this protocol in App. G (Hyperparameter selection) and in the Limitations.

For LoRA, the three ranks we tried show its trade-off between adaptation and retention, and none of them matches the MA-Conformer on both ASR and text (Table 9). This rank sweep was already reported in the appendix of the submitted version. The learning rate of 1e-4 is within the standard range for LoRA. For example, QLoRA uses the same learning rate for its 33B and 65B models [6]. In addition, LoRA does not fall short because it learns too little. At r = 356, it already reaches WERs of 3.2 and 7.1, but it loses more text ability than full fine-tuning (Text Δ −35.6 vs. −32.5). A lower rank forgets less but increases WER (6.2 and 15.8 at r = 16). Biderman et al. [7] also report that LoRA and full fine-tuning lie on a similar learning–forgetting curve. In the image setting, LoRA keeps the text ability at almost the same level as our method, but its Vision Avg. is much lower (49.8 vs. 54.9 at 1.7B and 54.1 vs. 58.7 at 3B).

To examine whether the learning rate causes the gap in the speech setting, we additionally trained full fine-tuning on SmolLM2-1.7B with a peak learning rate of 1e-5, keeping all other settings unchanged. The table below compares it with the runs in Table 1, and we have added this result to Table 9.

MethodPeak lrTrainableWER clean ↓WER other ↓Text Δ
Full fine-tuning1e-41.87B2.35.6−32.5
Full fine-tuning (new)1e-51.87B2.87.2−27.5
MA-Conformer (matched)1e-40.66B2.35.1−6.8
MA-Conformer1e-40.81B2.35.1−6.9

As shown in the table, lowering the learning rate reduces forgetting but increases WER. Therefore, full fine-tuning moves along the same kind of trade-off as the LoRA ranks in Table 9. At neither of the tested learning rates does full fine-tuning approach the trade-off of the MA-Conformer (WER 2.3 and 5.1, Text Δ −6.9).

Uncertainty of small gaps

As noted in the Reproducibility Statement, all results come from single runs. However, the direction of the main comparisons is the same across two scales, three settings, and several separately trained controls (the parameter-matched variants, the 1D-kernel variant, bidirectional attention, and the 1×1 kernel). In addition, to obtain a statistical estimate without retraining, we computed 95% confidence intervals for the WER reduction from transformer expansion to the MA-Conformer, using a paired bootstrap over LibriSpeech utterances [8]. The reduction in absolute WER points is significant on test-other at both scales (0.4 [0.1, 0.7] at 1.7B and 0.4 [0.1, 0.7] at 3B) and on test-clean at 3B (0.3 [0.1, 0.5]). The interval includes zero only for test-clean at 1.7B (0.1 [−0.1, 0.3]), where both WERs are already low (2.4 vs. 2.3). These intervals reflect the variability over test utterances for the trained checkpoints, and they complement the consistent direction of the results across scales, settings, and controls described above. We have added these intervals to App. F (Table 8) and referred to them in Section 3.1.

Reviewer_fZqJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends a recent paper by Yano et al. to adapt a pre-trained LM to new modalities by adding adaptor layers. New layers are inserted into the pre-trained model, which is frozen. Speech and image inputs use 1D/2D convolutional layers respectively but otherwise parameters are shared. To allow different speed/accuracy trade offs, dropout is applied to the adaptor layers during training. Results show the model can achieve good performance on multimodal task.

Critical strengths

Adapting text LLMs to other modalities is an important problem, and the paper introduces a new method that achieves good results

Critical weaknesses

The paper focuses on extending previous work to support both image and speech, but I couldn't find a clear articulation of why doing this with the same adaptor is desirable. Why not Relatedly, I couldn't see any evidence of any cross modality transfer between speech and image that might justify this approach. Appendix G shows that the method can cause very large degradations on generative text tasks. This seems like an important result for understanding the effectiveness of the method, so should be discussed more prominently. While the results tables compare methods by trainable parameters, it would also be good to include the compute cost (which I think is significantly higher than for the baseline) Overall the contributions on top of Yano et al (2026) feel quite thin.

Questions for authors

None

3 comments

Thank you for your review. We would like to address your concerns. We have uploaded a revised version of the paper, and all section, table, and appendix numbers below refer to the revised version.

Weakness 1: Shared expansion layers

We interpret this concern as: why do speech and images go through the same expansion layers instead of separate adapters? To be clear, the layers are only partially shared. Each layer has its own cgMLP per modality. Only the MHSA, FFN, and RMSNorm layers are shared (Section 3.3.1). We shared these parts to keep the cost low for each new modality. The cgMLP adds 0.15B per modality at 1.7B (Tables 2 and 4). Copying the expansion layers entirely would add about 0.55B instead (0.81B in Table 2 minus the 0.26B embedding and output matrices included in Trainable, App. A). The shared model can also read inputs containing both modalities. Localized Narratives is one example: each image has a spoken description. Freezing one backbone for multiple modalities is also used in practice. For example, Phi-4-Multimodal freezes the language model and adds LoRAs for each modality [1]. We have added this motivation to the introduction (Section 1).

Weakness 2: Cross-modal transfer

The review notes the lack of “evidence of any cross modality transfer between speech and image.” We would like to clarify that our paper does not claim cross-modal transfer. As described in the Limitations, we report the trimodal results for each modality separately (Section 3.3). In addition, the reasons for sharing the layers in Weakness 1 do not depend on transfer. We also checked whether the shared model uses one modality when it processes the other modality. This result is already reported in the paper (Section 3.3.2 and App. E). In our trimodal data, Localized Narratives accounts for 40% (Table 6), and each image is paired with a spoken description. On 1k held-out examples, giving the image to the model reduces the WER of the description from 10.0 to 9.7 (Table 7). We consider this as a property of the shared model, not as evidence of transfer. In each expansion layer, the shared MHSA directly mixes information across the modality spans, while the cgMLPs operate separately within each modality (Equations 3 and 4). The purpose of our trimodal setting is to investigate whether depth up-scaling can give a single model the ability to process multiple modalities. We think that the verification of cross-modal transfer is an important direction for future work.

Weakness 3: GSM8K degradation

In the submitted version, we placed the GSM8K results in the appendix because of the page limit and described the degradation in the Limitations. Since an additional page is now allowed, we have moved these results to Section 3.3 (Table 5). In the image setting, depth up-scaling almost keeps the performance of the base model, while full fine-tuning clearly degrades it (Table 5). In the speech setting, the performance of all methods decreases. However, depth up-scaling still keeps higher performance than full fine-tuning (16.1 vs. 1.2). In addition, for text-only inputs, we can skip the expansion layers and remove the added vocabulary entries, and then the original performance is recovered exactly. We also evaluated the trimodal setting during the rebuttal period. In this setting, the MA-Conformer keeps 55.8, while full fine-tuning drops to 12.2 (Table 5, see also our response to Reviewer RXXq, Weakness 5).

We think that one likely reason for the degradation in the speech setting is the training data. The text side of the speech data consists of short ASR transcripts. As reported in SALMONN [2], such data causes speech LLMs to over-fit to the task. More fundamentally, this paper focuses on continual pre-training. Its output is intended as a starting point for supervised fine-tuning and other post-training, not as a final model. Therefore, the important question is whether it is still an appropriate starting point. On knowledge benchmarks, our method is much better than full fine-tuning (Table 1). After supervised fine-tuning, the MA-Conformer outperforms transformer expansion on all four image benchmarks, as already reported in the appendix of the submitted version (App. K). Even when the few-shot generation performance decreases, depth up-scaling preserves much more of the base model than full fine-tuning. From this perspective, we believe that our method provides a better starting point for post-training than full fine-tuning and LoRA.

Weakness 4: Compute cost

The review assumes that the computational cost will be “significantly higher than for the baseline.” We would like to clarify that the training cost was already reported in the appendix of the submitted version (App. I, Table 10 in the revised version), and the cost of our method is at the same level as that of the baselines. The table below shows the training cost for speech-to-text on SmolLM2-1.7B. All methods use the same number of steps and the same hardware.

MethodTrainableGPU-hours
Full fine-tuning1.87B116
LoRA (r = 356)0.66B131
Depth up-scaling (Transformer)0.66B97
Depth up-scaling + MA-Conformer0.81B124

As shown in the table, the MA-Conformer requires 124 GPU-hours. This is only 7% more than full fine-tuning (116 GPU-hours), and it is even 5% fewer than LoRA (131 GPU-hours). Compared with transformer expansion, it requires 28% more because of the convolutional branch. The difference between the methods is small because every method runs the forward and backward passes through the whole base model at each step, and this part dominates the cost. In addition, since depth up-scaling freezes the base model, it does not need weight gradients or optimizer states for the base parameters. As a result, the peak memory is 23% lower than that of full fine-tuning (45.3 GB for the MA-Conformer vs. 58.6 GB per GPU). Because this comparison uses the same batch size for all methods, this memory headroom may allow a larger batch size for depth up-scaling. From the above, the training cost of our method is not significantly higher than that of the baselines.

Weakness 5: Novelty

We respectfully disagree that our contributions on top of Yano et al. [3] are thin. To the best of our knowledge, this paper is the first to show the following three points. First, the local operation of the expansion layers in a frozen LLM should follow each modality (Tables 2 and 3). Second, one shared set of expansion layers can serve both speech and images (Table 4). Third, among the drop schedules we compare, only Ordered Drop, which trains on the same nested subsets used at inference, avoids the collapse at small depths (Section 3.4). Yano et al. studied speech only, and these questions were left open. Our findings fill this gap. In addition, during the rebuttal period, we added controls with a 1×1 kernel and with bidirectional attention within the image tokens. Neither of them improves on transformer expansion, while the 1D and 2D kernels do (Table 3, see also our response to Reviewer RXXq, Question 1). This result shows that the gain cannot be explained only by added capacity or gating, and that local mixing that follows the token structure is important. We believe that these findings provide concrete design principles for practitioners who adapt frozen LLMs to new modalities.

References

[1] Abouelenin et al. 2025. Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs.
[2] Tang et al. 2024. SALMONN: Towards Generic Hearing Abilities for Large Language Models. ICLR 2024.
[3] Yano et al. 2026. Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling. Interspeech 2026.

As the discussion period is coming to an end, we appreciate your review. We believe that our response and the revised version have addressed the concerns you raised. We would appreciate it if you could let us know if you still have any concerns or if there are any remaining points that need further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.