acceptodds
Withdrawn from ICLR 2027

Sequence Matters: A Controlled Study of Modality Injection Order in Multimodal LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) integrate heterogeneous inputs such as text, vision and audio and are increasingly fine-tuned as pretrained backbones across various downstream classification tasks. While recent studies suggest that the ordering of multimodal inputs may affect model behavior, existing work remains largely limited to isolated settings, focusing on a single dataset, task or model. We introduce a Modality Order-aware Controlled Architecture for MLLMs namely , that enables controlled multimodal scheduling during decoder processing. This work is positioned as a controlled empirical study rather than a mechanistic interpretability analysis. Building upon this framework, we conduct a large-scale empirical study on the impact of modality injection order during fine-tuning, across four MLLMs, six downstream tasks, twelve multimodal datasets, three modalities and six modality orders. Our results reveal consistent and task-dependent performance patterns with regard to modality order, demonstrating that the choice of injection sequence can substantially influence fine-tuning outcomes under the present controlled setting. These findings reveal modality order as an empirically consequential yet underexplored design variable in downstream multimodal adaptation and provide practical guidance for future MLLM fine-tuning. Furthermore, comparisons with an All-at-Input baseline reveal that simply increasing the processing depth of all modalities does not consistently improve performance, suggesting that modality scheduling merits careful consideration in downstream adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Modality Injection Order, Multimodal Fine-tuning, Empirical Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tLTeOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MOCA-MLLM, a framework for studying modality injection order during multimodal LLM fine-tuning. The work aims to examine whether injecting text, vision, and audio at different decoder depths affects downstream performance. The authors first extract modality features using frozen text, vision, and audio encoders. Then they project and sequentially inject these features at three decoder depths according to all six possible modality orders. They further compare sequential injection with an All-at-Input baseline. Experiments on four MLLM backbones, six classification tasks, and twelve datasets report task-dependent performance differences among injection orders.

Critical strengths
  1. The motivation of this work is inspiring: whether modality injection order affects downstream MLLM adaptation.
  2. The writing style is good and easy to follow.
Critical weaknesses
  1. The experimental setting is confusing. The evaluated MLLMs already have pretrained modality encoders and connectors aligned with their backbones, but MOCA replaces them with external MiniLM, CLIP, and Wav2Vec2 features and newly added projections. Since these representations were not jointly pretrained with the backbones, it is unclear whether the original multimodal capabilities remain valid; the results may reflect representation-space mismatch rather than modality order. Comparisons using the native encoders are needed.
  2. The paper is highly similar to "Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models." Although this work is cited, the main extension appears to be moving from inference-time context ordering to depth-wise modality injection during fine-tuning. The authors should clearly distinguish the two works and justify why the contribution is more than an incremental extension.
  3. The comparison is limited to All-at-Input. The paper lacks comparisons with native MLLM fine-tuning methods, such as LoRA, making it difficult to assess the practical value of MOCA-MLLM.
  4. The injection layers are fixed at approximately L/3 and 2L/3. Simply dividing the decoder into three equal parts may not be optimal. Ablation studies with different injection depths and spacing strategies are required.
Questions for authors

See weaknesses. If the authors can adequately address these concerns, especially weakness 1 and 2, I would consider raising my score.

Reviewer_ec9DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a controlled empirical study of how modality injection order affects MLLM fine-tuning. It investigates different sequential injection orders of text, vision, and audio at different decoder depths. Experiments cover four MLLM backbones, six tasks, and twelve datasets. The results show that the effectiveness of modality order varies across downstream tasks, with certain injection sequences outperforming the baseline. In summary, this study provides empirical evidence for the impact of modality scheduling on downstream performance.

Critical strengths

The paper conducts extensive experiments across multiple tasks, datasets, and MLLM backbones, providing broad empirical validation of the observed modality-order effects. Evaluating diverse settings strengthens the credibility of the findings and makes the study more informative.

Critical weaknesses

While the paper provides extensive empirical evidence of modality-order effects, the analysis could be strengthened by exploring the underlying reasons for these observations. For example, examining how different injection orders affect cross-modal interactions and attention to specific visual or audio features could provide valuable insights into the observed performance differences. Such analyses would help explain why certain configurations work better for particular tasks.

Questions for authors

The paper emphasizes the practical value of MOCA-MLLM for downstream adaptation. However, existing studies have reported substantially higher performance for some backbones, such as LLaVA-1.5, than the All-at-Input baseline in this paper. Could the authors clarify this discrepancy and evaluate MOCA-MLLM under standard fine-tuning settings for these backbones to better establish the practical benefits of the proposed modality injection strategy?

Reviewer_tGrZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper finds that, during fine-tuning for downstream multimodal classification, appropriately staged injection of text, vision, and audio often outperforms introducing all three modalities at the decoder input. Through comparative experiments across tasks, it investigates the depth and order at which different modalities are introduced. The results show that the best staged injection configuration outperforms the All-at-Input baseline in most settings and provide task-specific empirical observations.

Critical strengths
  1. Clear and practically relevant research question. The paper examines the timing and depth of modality injection, an often-overlooked design dimension in downstream multimodal fine-tuning.

  2. Broad experimental coverage. The evaluation covers four backbones, six task categories, and twelve dataset evaluation settings, with results averaged over twenty runs, lending credibility to the empirical findings.

  3. Useful empirical observations from systematic comparisons. By comparing all six modality permutations against the All-at-Input baseline, the paper finds that the best staged injection configuration achieves higher accuracy in most settings and identifies patterns across different scenarios.

Critical weaknesses
  1. The paper primarily presents an empirical summary and reads more like an experimental report. It lacks theoretical grounding or an in-depth analysis of the underlying causes that could strengthen its scientific depth and generalizability. Its reliance on aggregating extensive experimental results substantially limits its scientific contribution and novelty.
  2. The experiments rely on external encoders, some pre-extracted features, and preprocessing choices such as static-frame selection. It remains unclear whether the findings transfer to native multimodal input pipelines or generative tasks. Furthermore, exhaustive search provides only an empirical reference; it does not yield an efficient configuration selection strategy or broadly applicable guidance.
Questions for authors
  1. Is there a general method or metric that can efficiently determine suitable modality injection depths and timing?
  2. Could the authors add experiments that vary injection depths while holding modality order fixed, as well as permutation controls in which all modalities are introduced at the same layer, to disentangle the contributions of modality order and processing depth?
  3. Do the task-level observations transfer to previously unseen datasets that were not used to derive them? Were the best modality orders reported in the tables selected exclusively based on validation performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.