acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Is Not Enough: Understanding Generalization in Multimodal Fusion

Abstract

Modality imbalance often prevents multimodal learning from fully exploiting the information available in each modality. Although decoupling unimodal learning from fusion alleviates optimization interference, it does not ensure fusion generalization. We identify a remaining source of failure: fusion trained on samples used for unimodal fitting can exploit modality reliability and error patterns that transfer poorly to unseen data. We analyze this failure mechanism and propose a generalization-aware multimodal fusion framework. The framework constructs out-of-fold representations and predictions to learn feature-based corrections to averaged unimodal logits from out-of-sample evidence. To account for differences among the unimodal models fitted across folds, we formulate a cross-fold objective that balances average correction gains against unfavorable fold-specific effects. Experiments across multimodal benchmarks support the effectiveness of the framework, with component ablations demonstrating the complementary benefits of out-of-fold fusion learning and cross-fold regularization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal imbalance learning, multimodal fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_guDaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a potential source of generalization failure in multimodal fusion that persists even when unimodal representation learning and fusion optimization are decoupled. The authors argue that fusion modules trained on samples already used for unimodal fitting may learn modality-specific reliability and error patterns that do not transfer well to unseen data.

To investigate this issue, the paper presents an analysis of how fusion optimization can reinforce unimodal decision directions that fit training samples but generalize poorly. Based on this observation, the authors propose an out-of-fold residual fusion framework, where unimodal models are trained on complementary folds and a shared fusion head learns corrections to averaged unimodal logits using out-of-sample representations. A cross-fold robust regularization objective is further introduced to discourage corrections that cause unfavorable performance changes on individual folds.

Experiments on seven multimodal classification benchmarks demonstrate competitive results against recent multimodal learning methods and independent unimodal ensembles. Additional ablations investigate the contributions of out-of-fold training, residual fusion, and cross-fold regularization.

Critical strengths
  1. The paper highlights an important limitation of decoupled multimodal learning: eliminating optimization interference does not necessarily prevent a fusion module from learning poorly generalizable relationships between modalities.

  2. The framework is evaluated on seven benchmarks spanning audio-visual, image-text, and trimodal classification. The proposed approach achieves strong results, particularly on CREMA-D and UR-FUNNY.

  3. The narrative is clear and the language is well-articulated.

Critical weaknesses
  1. The proposed framework combines out-of-fold unimodal predictions, a learnable residual correction to averaged logits, and a worst-fold regularization term. While this combination is reasonable, its central ingredients are closely related to established cross-fitting/stacking procedures and group-robust optimization objectives.

  2. Several improvements over the strongest competing methods are relatively small, particularly on AVE, MVSA, UPMC-Food101, and CMU-MOSEI.

  3. Some reported accuracy values warrant clarification regarding the evaluation split and aggregation protocol. For example, Table 5 lists 402 validation and 401 test samples for AVE, whereas the reported Ours accuracies of 65.17%, 39.05%, and 71.64% are numerically consistent with integer correct-prediction counts over 402 samples. Similarly, some MVSA results align with its reported validation-set size. They may arise from averaging across runs but there is no information about this. The paper does not provide a sufficiently detailed computation-performance comparison with methods using comparable training and inference budgets.

  4. Figure 3 has a somewhat generic, infographic-like appearance, with excessive decorative elements that detract from the clarity expected of a technical illustration. More importantly, many labels and mathematical annotations are too small to read comfortably at normal viewing scale, making the overall workflow difficult to follow.

  5. The central mechanistic analysis, including the classifier-direction projections and component-removal intervention, is primarily conducted on CREMA-D with an affine fusion head. While the proposed method is evaluated on seven datasets and several fusion architectures, it remains unclear whether the identified mechanism consistently explains generalization failures in other settings, particularly nonlinear fusion and trimodal learning.

Questions for authors

See weaknesses. If the authors can provide a clear explanation, I will consider raising my score.

Reviewer_CYAVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies generalization failures in decoupled multimodal learning caused by discrepancies between in-sample and out-of-sample modality reliability. The authors propose an out-of-fold residual fusion framework with cross-fold robust regularization to improve fusion generalization. Experiments on multimodal benchmarks demonstrate its effectiveness.

Critical strengths
  1. The paper identifies an important generalization issue in decoupled multimodal learning, supported by theoretical analysis and controlled experiments.

  2. Extensive experiments across seven benchmarks demonstrate competitive performance.

Critical weaknesses
  1. The proposed Out-of-Fold (OOF) residual fusion framework primarily addresses the in-sample versus out-of-sample generalization gap in unimodal predictions, rather than directly tackling modality imbalance. However, the paper lacks sufficient discussion of and comparisons with existing methods addressing such generalization gaps, making it difficult to assess the novelty and specific advantages of the proposed framework.

  2. The framework requires training and evaluating multiple unimodal models, introducing additional computational overhead.

Questions for authors
  1. Could training unimodal models on only a subset of the data alter their relative generalization performance and reliability, potentially leading the fusion module to learn suboptimal modality relationships? Could the authors provide further analysis of this effect?

  2. Could the authors clarify whether the fold-specific performance differences arise mainly from variations in unimodal predictors or differences in sample composition, and why minimizing the worst-fold loss improves fusion generalization?

Reviewer_UbpiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors observe that unimodal training accuracy may not be indicative of generalisation, especially in the multimodal setting when the model has a choice of adaptively leveraging multiple modalities through fusion. Subsequently, their claim is that unimodal and multimodal training should happen on separate subsets of a sample so that unimodal generalisation failure modes that might have been obscured by modality-specific overfitting can be resolved through the process of training the fusion head.

Critical strengths
  1. The observation that unimodal training and multimodal fusion could have different modes of generalisation is interesting.

  2. The claims made with regards to better generalisation via out-of-fold fusion head training may be significant but the current experimental setup has significant gaps, resolving which remains necessary before the paper can be evaluated on methodological grounds.

Critical weaknesses
  1. One of the most significant concerns regarding the work is its experimental evaluation protocol. The authors generally do not clearly discuss the number of unique samples used in the different settings. In fact, reading from lines 49 to 53 in the Introduction, which forms the basis for Fig 1(b), it seems that the in-sample fusion setting never sees 10% of the dataset that the held-out fusion setting sees during training the fusion module. This is a serious issue of mismatched training datasets and I suspect a significant proportion of the gains reported in this paper comes from this discrepancy.

  2. The reported results lack uncertainty estimates, and the manuscript does not specify the number of independent runs or random seeds. It is therefore difficult to assess the reliability of the improvements, particularly differences below one percentage point. The authors should report the number of runs and variability across seeds, clarify whether the reported values are individual runs or averages, and assess sensitivity to fold assignment for the central ablations. Table 1 also states that standard deviations from previously reported results were omitted; retaining these would help readers assess the comparisons.

  3. The above two points are very basic experimental hygiene that need to be followed and the fact that they have been ignored sets a rather unfavourable precedent for the paper and possibility of more such crucial errors. I would advise the authors to thoroughly revisit their end-to-end protocols and ensure that aspects related to basic setup are correctly implemented - otherwise, the ideas risk getting rejected at the very outset.

Questions for authors

Refer to the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.