acceptodds
Under review as a conference paper at ICLR 2027

Mixture-of-LoRA: Modality-Specialized Low-Rank Adapters for Multimodal Language Models

Abstract

Adapting pretrained Large Language Models (LLMs) to process multimodal inputs typically requires either costly full fine-tuning or training from scratch on trillions of tokens. We observe that even when a shared low-rank adapter is used for multimodal fine-tuning, latent representations cluster by modality across layers—suggesting that modality-specific adaptation is a natural inductive bias. Motivated by this finding, we introduce Mixture-of-LoRA (MoL), a parameter-efficient fine-tuning framework that injects per-modality LoRA adapters into the frozen attention layers of a pretrained LLM, while keeping global self-attention intact for cross-modal fusion. MoL brings the modality-specialized design of Mixture-of-Transformers (MoT) to the adapter setting, enabling multimodal understanding without modifying the base model's weights. We evaluate MoL on image-text (21 benchmarks) and audio-text tasks, comparing against shared LoRA, DoRA, modality-only adapters, and their mixture variants. MoL and its DoRA counterpart (MoD) consistently outperform shared adapters—achieving the highest average scores with especially large gains on counting, spatial reasoning, and chart understanding tasks. Our analysis further reveals that per-modality adapters produce sharper attention patterns and stronger cross-modal attention compared to shared adapters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal learning, Parameter-efficient fine-tuning, Low-rank adaptation, Large language models, Modality-specific adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_77ScOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses multimodal adaptation by replacing a single LoRA adapter shared across all modalities with modality-specific LoRA adapters, while retaining global self-attention for cross-modal fusion. The authors evaluate the approach using Llama 3.2-3B, training separate image–text and audio–text models.

Critical strengths
  1. Clear and meaningful research problem. Effective multimodal adaptation with a frozen pretrained LLM is a practically valuable research direction.
  2. Simple and intuitive method. MoL introduces modality-specific low-rank updates into the attention projections while preserving global self-attention, supporting both modality specialization and cross-modal interaction. It requires neither a learned router nor modifications to the backbone weights and extends naturally to DoRA.
  3. Broad experimental coverage. The paper evaluates the proposed method on 21 image–text benchmarks and three audio benchmarks, providing extensive empirical evaluation.
Critical weaknesses
  1. Limited methodological novelty. The core design primarily applies MoT’s modality-specific parameterization to LoRA, with adapters selected deterministically based on known modality identities. The paper should clarify its distinct technical contributions relative to existing multi-adapter and LoRA expert methods.
  2. Modest overall gains and substantial variation across tasks. Although MoL and MoD improve the overall average score, performance declines on several tasks. For example, MoL underperforms shared LoRA on the OCR category, CVBench, VSR, and TallyQA.
  3. Limited evidence of generalizability and scalability. Experiments are conducted only on Llama 3.2-3B and involve pairwise modality combinations. This leaves the method’s applicability to larger models and settings with more modalities insufficiently established.
Questions for authors
  1. Have the authors sufficiently reviewed existing work on LoRA-based multimodal adaptation? Could the relevant work be discussed more systematically, with clearer distinctions between the proposed approach and prior methods?
  2. The main text describes the computational overhead as negligible, whereas the appendix reports per-step latency increases of 28.0% and 40.2% for MoL and MoD relative to LoRA. Can the authors provide end-to-end timing results to substantiate this claim?
  3. When the image and text adapters use the same scaling factor α/r\alpha/r, does the image adapter still exhibit a larger update norm? Is this asymmetry associated with the observed performance gains?
Reviewer_xjaHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Mixture-of-LoRA (MoL), a PEFT-analogue of the Mixture-of-Transformers (MoT) model which adapts the attention layers of a frozen pretrained text LLM using modality-specific low-rank adapters. Like in MoT, despite functionally using different attention projection weights, the attention operation is global, retaining cross-modal interaction. The authors also explore an analogue Mixture-of-DoRA (MoD) based on the same principle but using DoRA adapters. MoL and MoD also use modality-specific normalization layers. The authors evaluate these approaches on a Llama3.2-3B model in image-text and audio-text settings, comparing them against shared LoRA/DoRA adapters and an image-only LoRA as baselines. The paper reports improvements in aggregate performance across benchmarks, and some experiments that examine the learned adapter magnitudes and attention patterns.

Critical strengths

None.

Critical weaknesses

I see a few significant issues with the paper:

  • Modality-specific low-rank multimodal adaptation of LLMs has been already extensively studied in the literature. A highly relevant example is MM-LoRA (Sun et al. 2024) which uses essentially an identical method for VLMs: separate LoRAs for language and vision inputs. Since their LoRA is applied to the projection matrices and otherwise the layers are executed normally, it also incorporates the idea of global attention for cross-modal interaction. Like in the present paper, the authors ablate using a shared adapter vs. per-modality adapters (Table 4), and already provide evidence towards the superiority of the latter. While this paper additionally shows the benefit in the text-audio domain, evaluations in different benchmarks, and applies it with DoRA adapters as well, this is, in my opinion, merely the validation of an already proposed and well established approach and does not merit the claim to a novel significant contribution. Furthermore, there are many other works investigating similar approaches over which this work contributes little novelty, among them: MoSS (Xu et al., 2025), which uses per-modality LoRA for image and text tokens; MokA (Wei et al., 2025), which studies per-modality LoRA and cross-modal interactions across visual and audio settings; and MoLoRA (Shah and Wagle, 2026), which uses per-token adapter routing, including in multimodal sequences.

  • The experiments do not isolate the benefits of modality-specific adapters. The main experiments introduce modality-specific normalization layers and use different scaling factors across modalities, and the effect of these is not isolated from that of the modality-specific LoRA/DoRA. The authors also do not mention how they calibrated their optimization hyperparameters, e.g. they do not explain why MoL adapters have a different minimum learning rate ratio.

  • The authors often over-claim results. In Table 1, MoL outperforms shared LoRA on only 8 of 21 image-text benchmarks, with an average improvement of approximately 1.44 points. Notably, 91% of this average improvement comes from CountBench, and excluding this benchmark reduces the gain to approximately 0.13 points. While the gain in CountBench is significant, the model regresses on other benchmarks from the same category (counting and spatial). The authors' claim that per-modality adapters improve counting and spatial reasoning is therefore not well supported. MoD exhibits somewhat more consistent gains, but its aggregate gain remains modest. Furthermore, no training-seed variability is reported. Similar over-claiming occurs with the analysis of the adapter norms, where the reported larger update for images can be at least partly explained by the image adapters being assigned twice the effective scaling factor of the text adapters. The analysis of the attention patterns is done in a very small sample and the observed patterns are largely descriptive, with no evidence showing they are connected to the downstream performance improvements. Finally, regarding computational costs, the authors claim negligible overhead, yet Appendix E shows training overheads of up to 40%. While the authors excuse this by stating the implementations are not optimized, the claim of negligible overhead should be substantiated somehow.

Questions for authors

I ask authors mainly to address the concern of lack of meaningful novelty. If a meaningful contribution can be substantiated, then to address the concerns regarding experimental design and analysis of results. However, if the novelty concern is not satisfactorily addressed, I would not consider raising my score.

And a question, rather out of curiosity: With MoL/MoD, do the representations from different modalities cluster differently from when using a shared adapter?

Reviewer_eHMyOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mixture-of-LoRA (MoL), a parameter-efficient fine-tuning method that assigns separate LoRA adapters to different modalities while preserving global self-attention for cross-modal fusion. Experiments on 21 image-text benchmarks and three audio-text benchmarks using Llama 3.2-3B show improved aggregate performance over shared LoRA and DoRA baselines.

Critical strengths
  1. Modality-specific updates provide a plausible way to accommodate heterogeneous encoder features while retaining a shared pretrained backbone and cross-modal attention. The method is straightforward and does not require a learned router. The connection to MoT is clearly acknowledged.
  2. The visual evaluation spans OCR, charts, VQA, counting, and reasoning. In Table 1, MoL improves CountBench from 33.73 to 61.31 and ChartQA from 53.32 to 60.72 over shared LoRA. Audio results in Table 3 provide additional evidence: MMAU increases from 0.376 for LoRA to 0.442 for MoL and 0.476 for MoD.
Critical weaknesses
  1. MoL and MoD introduce per-modality RMSNorm layers in addition to separate LoRA/DoRA adapters, whereas the shared-adapter baselines do not receive an explicitly matched normalization treatment. Consequently, the improvements cannot be attributed uniquely to modality-specific low-rank updates. A controlled ablation separating adapter specialization from normalization is necessary to validate the central claim.
  2. The image-text aggregate improvements over shared LoRA and DoRA are relatively modest, and several individual benchmarks show regressions. The paper reports no variance across independent training runs or uncertainty estimates.
  3. The use of in-house SFT datasets without sufficient information about their composition, size, and benchmark-overlap controls makes it difficult to assess reproducibility and potential data contamination.
Questions for authors
  1. Can the authors isolate modality-specific LoRA from modality-specific RMSNorm, scaling, and learning-rate schedules using matched factorial ablations? What are the exact trainable parameter counts for each component?
  2. How stable are the gains across training seeds? Can you explain the unusually large CountBench gain alongside regressions on TallyQA, VSR, and CVBench, and provide uncertainty estimates and benchmark-overlap checks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.