acceptodds
Under review as a conference paper at ICLR 2027

Strong-Modality Collapse in Multimodal Fusion: Causal Analysis and Structural Solution

Abstract

Multimodal fusion is expected to benefit from combining complementary signals. However, causal pathway isolation shows that standard symmetric cross-attention can degrade the predictive performance of the dominant modality, in some cases pushing the fusion model below its unimodal baseline. We call this failure strong-modality collapse. We find that it results from the interaction of two factors: cross-modal attention alters the dominant representation, while joint optimization adapts its classifier to these changes. In most configurations, neither factor alone causes substantial degradation, but together they produce losses of up to 42.6 percentage points. This finding motivates Inverted Asymmetric Fusion (IAF), which shields the dominant modality from cross-modal attention while allowing weaker modalities to attend to it. Because structural shielding alone cannot address initial feature imbalances, we introduce Modality-Aware Knowledge Distillation (MAKD) to strengthen weaker encoders before fusion. Across four multimodal benchmarks, IAF preserves the dominant pathway at its unimodal ceiling and improves multimodal performance over the strongest unimodal baseline.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Fusion, Modality Collapse, Cross-Modal Attention, Knowledge Distillation, Causal Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FbmdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies "strong-modality collapse," a phenomenon where symmetric cross-modal attention degrades the predictive accuracy of the dominant modality pathway during multimodal fusion. Using a 2×22\times2 causal decomposition, the authors attribute this degradation to the compounding interaction between cross-modal feature alteration and joint classifier adaptation. To address this, they propose a two-stage framework: Modality-Aware Knowledge Distillation (MAKD) to strengthen weaker encoders prior to fusion, and Inverted Asymmetric Fusion (IAF), which shields the dominant modality by bypassing cross-modal attention and freezing its unimodal classifier while allowing weaker modalities to attend to it. Experiments across four benchmarks (MultiHuSE, UPMC Food-101, UR-FUNNY, and MUStARD) validate the approach.

Critical strengths
  1. Pathway-level evaluation catches what fused metrics miss. 2.Empirical evaluation across four diverse datasets and multiple authentic pretrained backbones.
Critical weaknesses
  1. Preservation holds by construction. With frozen features, no attention and a frozen head, the pure pathway equals the unimodal model. Ceiling accuracy and CKA = 1.0 (Tables 6, 7, 9) are therefore guaranteed.
  2. The collapse may be benign. Under a summed-logit loss, heads co-adapt rather than act as standalone classifiers. On Food-101 BERT+PE-Core, the image pathway loses 42.6 pp, yet text rises to 85.38% and fused accuracy exceeds the ceiling. Symmetric Fusion falls below the ceiling only twice, both within one SD.
  3. Statistical support is thin. The interaction reverses on UR-FUNNY and vanishes for MUStARD E5+VideoMAE audio, and no confidence intervals are given. "Causal" overstates a five-fold ablation. No within-fold validation split is described for selecting hyperparameters or the dominant modality.
  4. Gains over simple baselines are weak. Without MAKD, IAF trails early or late fusion in five of eleven settings and Symmetric Fusion in three. On UR-FUNNY, most of the gain comes from MAKD (Symmetric+MAKD: 69.18%), and Δ uses the non-distilled ceiling (62.11% vs. 65.19%).
  5. Novelty is limited. Directional cross-modal attention [a] and language-guided asymmetric fusion [b] already exist. Fixing one branch while fitting the others resembles ReconBoost [c]. MAKD is well-studied crossmodal distillation (Hinton et al., 2015; [d]).
  6. SOTA comparisons are missing: OGM-GE [e], UMT/UME [f], MLA [g] and ReconBoost [c]. Tables 10 and 11 omit HKT [h] (77.36% UR-FUNNY, 79.41% MUStARD), and their baselines use older encoders.

[a] Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." ACL, 2019. [b] Zhang, Haoyu, et al. "Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis." EMNLP, 2023. [c] Hua, Cong, et al. "Reconboost: Boosting can achieve modality reconcilement." ICML, 2024. [d] Xue, Zihui, et al. "The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation." ICLR, 2023. [e] Peng, Xiaokang, et al. "Balanced multimodal learning via on-the-fly gradient modulation." CVPR, 2022. [f] Du, Chenzhuang, et al. "On uni-modal feature learning in supervised multi-modal learning." ICML, 2023. [g] Zhang, Xiaohui, et al. "Multimodal representation learning by alternating unimodal adaptation." CVPR, 2024. [h] Hasan, Md Kamrul, et al. "Humor knowledge enriched transformer for understanding multimodal humor." AAAI, 2021.

Questions for authors
  1. With fine-tuned encoders or token-level attention, does the dominant pathway still match its ceiling in accuracy and CKA?
  2. Does a linear probe re-fit on post-fusion dominant features recover the ceiling? Does the pathway drop correlate with fused accuracy across configurations?
  3. Please report confidence intervals for the interaction term per configuration, explain the UR-FUNNY reversal, and state how hyperparameters and the dominant modality were chosen within folds.
  4. Does IAF+MAKD beat Early/Late Fusion+MAKD with identical teachers, and how does Δ change against distilled ceilings?
  5. What is IAF's fused accuracy without CMA, i.e., a frozen dominant head plus a trained residual late-fusion branch?
1 comment

We thank the reviewer for the detailed review. Our responses follow:

W1 (preservation by construction): Yes, this follows from the design (Section 3.3): with frozen features, no attention and a frozen head, the IAF dominant pathway is the unimodal model, so ceiling accuracy and CKA = 1.0 are expected. What does not follow from the design is the fused accuracy, where IAF+MAKD is above the unimodal ceiling in all 11 configurations (Table 5), and the live-encoder check (Appendix F, one fold), where the text pathway is 2.70 pp below its ceiling with a frozen head, versus 9.54 pp for Symmetric Fusion.

W2 (collapse may be benign): Symmetric Fusion is below the unimodal ceiling in 2 of 11 configurations, both within one SD (Table 2). On Food-101 BERT+PE-Core, the image pathway falls from 85.46 to 42.89 while the text pathway rises from 76.08 to 85.38, and fused accuracy is 90.68. We will reword the claim as a pathway-level result that does not always lower fused accuracy. The pathway still matters when modalities are missing: in Appendix G, IAF matches or beats Symmetric Fusion in all 12 MultiHuSE comparisons where at least one non-dominant modality is present.

W3 (statistical support): The interaction term (Eq. 4) is above 3 pp in 10 of 15 pathway rows. It is near zero or negative in five: four MUStARD rows and UR-FUNNY, which we report as an exception in Section 4.3. “Most MUStARD configurations” should read “half”. We will replace “causal” with “factorial ablation”, add confidence intervals, and describe the validation split used to choose hyperparameters and the dominant modality.

W4 (gains over simple baselines): IAF is evaluated together with MAKD, so the fair comparison is IAF+MAKD against Symmetric+MAKD with the same teachers, where IAF is higher in 10 of 11 configurations (Table 5). On UR-FUNNY, MAKD accounts for most of the gain (Symmetric+MAKD 69.18, IAF+MAKD 69.80). Measured against the distilled dominant model, IAF+MAKD still gains +2.95 and +4.53 on MultiHuSE with BERT and +4.61 on UR-FUNNY. Elsewhere, the dominant model is the teacher, so the ceiling is unchanged.

W5 (novelty): MulT [a] applies directional cross-modal attention to every target modality, so no modality is shielded. IAF shields one modality, chosen from unimodal accuracy for each dataset (audio and video on MUStARD, where text is not dominant), and freezes its classifier, which the 2x2 ablation identifies as the second factor in the collapse. ReconBoost [c] alternates training so that one modality is updated while the others stay fixed, while IAF trains all pathways jointly with a structural shield. We will cite and discuss [b] and [c]. MAKD uses standard distillation losses (Eq. 7); it differs in placement before fusion and in using the dominant modality as teacher (Appendix M).

W6 (comparisons): OGM-GE, UMT/UME, MLA and ReconBoost were not run, and we will add them on our features where feasible. We will add HKT to Tables 10 and 11: it reports 77.36 on UR-FUNNY (IAF 69.80) and 79.41 on MUStARD (IAF 83.33). HKT fine-tunes ALBERT and uses external humor knowledge, which our setup does not. On UR-FUNNY we use the benchmark’s original features, for consistency with prior work. On MUStARD we use newer frozen encoders than the published baselines, so that comparison is indicative only, and we will state this in Appendix E.

Q1: With a fine-tuned encoder, the dominant pathway does not exactly match its ceiling. In the one fold we tested (Appendix F, LoRA on the text encoder), the text pathway was 2.70 pp below its ceiling with IAF’s frozen head, versus 9.54 pp for Symmetric Fusion. CKA was not computed there, and token-level attention was not tested. Both will be listed as limitations.

Q2: We did not run the linear-probe refit, and we will add it. On the correlation, a Spearman test computed from Tables 2 and 3 shows that the drop in the dominant pathway does not track fused accuracy across the 11 configurations (rho = -0.22, p = 0.52). For example, on Food-101 BERT+PE-Core, the image pathway falls by 42.6 pp while fused accuracy is 5.2 pp above the ceiling.

Q3: We will add confidence intervals for the interaction term in every configuration. On UR-FUNNY, classifier drift alone gives 48.31, which is chance for this binary task, and attention raises it to 57.83. The dominant modality is the one with the highest standalone validation accuracy (Eq. 1), and hyperparameters were tuned manually on validation performance (Appendix O). We will describe the validation split within each fold.

Q4: Early and Late Fusion with MAKD were not run, and we will add them with identical teachers. The only same-teacher comparison in the paper is with Symmetric Fusion (Table 5). Against the distilled dominant model, IAF+MAKD gains +2.95 and +4.53 on MultiHuSE with BERT and +4.61 on UR-FUNNY, versus -0.79, +1.79 and +3.99 for Symmetric+MAKD.

Q5. Not run, and we will add it.

Reviewer_bSFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies “strong-modality collapse,” in which symmetric cross-attention fusion degrades the dominant modality’s own pathway, and uses a 2×2 causal decomposition to show that this comes from the interaction between cross-modal attention and classifier drift. To address it, the authors propose Inverted Asymmetric Fusion (IAF), which shields the dominant modality from cross-attention and freezes its classifier, along with Modality-Aware Knowledge Distillation (MAKD), which strengthens the weaker modalities before fusion.

Critical strengths
  • The 2×2 decomposition, crossing cross-attention (on/off) with classifier state (frozen/trainable), is simple and easy to reproduce.
  • IAF needs no gradient rebalancing or extra loss terms, just a structural change. It also uses fewer parameters and less compute than symmetric fusion.
Critical weaknesses
  • The paper doesn’t compare IAF against the most direct baseline: a weighted ensemble of the frozen strongest unimodal model and separately trained models for the weaker modalities.
  • IAF is structurally close to gated late fusion, and the paper does not separate its contribution from ensembling.
Questions for authors
  • IAF keeps a frozen, complete unimodal model (original representation plus original classifier), and the gate can put its weight on that pathway. So IAF is essentially an ensemble of the frozen strongest model and the weaker modalities. Also, by the authors’ own metric, IAF just shifts the degradation onto the weaker modalities. In Table 6, the MultiHuSE audio pathway drops from 63.7 to 48.4, and the Food-101 text pathway drops from 85.78 to 73.39.
  • The paper doesn’t compare against modality imbalance methods such as OGM-GE (https://arxiv.org/pdf/2410.11582), PMR, AGM, and MMPareto on the same features. It also doesn’t compare against the simplest baseline: a weighted ensemble of the frozen strongest unimodal model and the weaker modality models, which is the most direct comparison for IAF. In addition, early fusion (95.42) actually beats IAF on Food-101.
2 comments

We thank the reviewer for their careful evaluation and constructive feedback. We address each comment below:

W1, W2 (ensemble baseline): We did not compare against a weighted ensemble of the frozen dominant model and separately trained weaker models. We agree it is the closest baseline. The only difference between it and IAF is that IAF's weaker branches attend to the dominant modality. We will add it.

Q1 (degradation moves to weaker modalities): The drops you cite (63.7 to 48.4, and 85.78 to 73.39) compare Symmetric and IAF pathways, not the unimodal accuracy of each modality. Against unimodal accuracy (Table 8), IAF’s MultiHuSE audio pathway is lower in 3 of 4 configurations (equal in the fourth), and the Food-101 text pathway is 1.9 pp lower (73.39 versus 75.3 for distilled BERT). So IAF does not preserve weak pathways. We defined collapse only for the dominant modality, because it is the one the model should be able to fall back on, and we judge weak modalities by what they add: removing either one lowers full IAF accuracy (Table 16). We will add the unimodal reference for the weak modalities to Table 6 and state this limitation in the text.

Q2 (baselines): AGM is in Table 10. We did not run PMR, MMPareto, OGM-GE, or the ensemble baseline, but we will add them where feasible. Early fusion does beat IAF on Food-101 (95.42 versus 94.72 for IAF and 94.27 for IAF+MAKD, Tables 2 and 5). The two modalities differ by only 0.78 pp there, so there is no strong modality to protect, and we do not claim IAF helps on that dataset.

my main concerns are still open. Since the response has only promises and no new results, I will keep my rating

Reviewer_r6xeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the degradation of a dominant modality's predictive performance after multimodal fusion, which the authors term strong-modality collapse. To characterize this phenomenon, the authors employ pathway isolation, causal decomposition, and representational similarity analysis to examine the effects of cross-modal attention and classifier adaptation on the dominant modality's prediction pathway. These analyses suggests that the interaction between these two factors can lead to substantial performance degradation. Building on these findings, the paper proposes a two-stage approach comprising Modality-Aware Knowledge Distillation (MAKD) and Inverted Asymmetric Fusion (IAF). MAKD uses dataset-specific teacher configurations to improve the representations of weaker modalities, while IAF prevents the dominant modality from receiving cross-modal attention and limits classifier adaptation along its prediction pathway, allowing weaker modalities to exploit information from the dominant modality. Experiments on four multimodal benchmarks, MultiHuSE, UR-FUNNY, MUStARD, and Food-101, compare the proposed approach with Early Fusion, Late Fusion, and Symmetric Fusion to evaluate its effectiveness.

Critical strengths
  1. The paper investigates an important issue in multimodal fusion, namely the degradation of a dominant modality's predictive performance after fusion, which can be overlooked when evaluation focuses on the final fused prediction. Through pathway isolation and controlled experiments, the paper examines where this degradation occurs and how it relates to cross-modal attention and classifier adaptation, providing a more detailed analysis of strong-modality collapse.
  2. The design of IAF is well aligned with the preceding analysis. Specifically, it prevents cross-modal attention from modifying the dominant modality's representation and limits classifier adaptation, while allowing weaker modalities to exploit information from the dominant modality. Moreover, IAF exceeds the unimodal ceiling in all 11 evaluated configurations, providing empirical evidence of the effectiveness of this asymmetric fusion strategy across the evaluated datasets and modality configurations.
Critical weaknesses
  1. The overall framework diagram is poorly laid out. The arrangement of modules and connecting arrows does not clearly convey the relationships among components or the overall workflow, making the proposed method difficult to follow.
  2. The implementation of MAKD is insufficiently clear given its stated goal of enhancing weaker modalities. Although the pretrained encoders remain frozen during both distillation and fusion, the trainable components are not clearly specified. Moreover, the dual-teacher formulation for MUStARD is ambiguous, as T1T_1 is defined as the primary teacher, yet w1w_1 varies with the dominant modality. MAKD also reduces unimodal accuracy for some encoders on Food-101, while adding MAKD to IAF lowers fusion accuracy from 94.72% to 94.27% for E5+PE-Core, indicating inconsistent gains across configurations.
  3. The operational definition of strong-modality collapse captures pathway-level predictive degradation but does not establish modality-specific information loss. The paper identifies collapse based on Accpath<AccuniAcc_{path}<Acc_{uni}, yet a decline in single-path accuracy does not necessarily indicate that task-relevant information has been lost. Moreover, CKA measures representational similarity rather than task-relevant information loss. In IAF, the dominant modality bypasses cross-modal attention, so the CKA of 1.0 reported for this pathway in Table 9 follows directly from the model architecture and provides limited independent empirical evidence.
  4. For MultiHuSE and MUStARD, “BERT+VideoMAE” still includes the fixed Dasheng audio encoder, while “GloVe+OpenFace” in Table 6 omits the COVAREP audio features used for UR-FUNNY. Consequently, the configuration names do not fully reflect the modalities involved in fusion.
  5. Section 4.3 uses controlled experiments to examine the interaction between cross-modal attention and classifier adaptation, but the source of the frozen classifier parameters and the update strategies for other model components remain unclear. Moreover, on UR-FUNNY, classifier adaptation alone reduces accuracy by 13.80 pp, whereas enabling cross-modal attention increases it from 48.31% to 57.83%. This suggests that the interaction between these two factors varies across datasets and does not consistently exacerbate performance degradation.
  6. The main experiments focus on Early Fusion, Late Fusion, and Symmetric Fusion, without direct comparisons against gradient modulation, Pareto-based optimization, and prototype-guided methods under consistent experimental settings. Although Appendix E reports results for some existing methods on three datasets, MultiHuSE is excluded due to differences in evaluation protocols. Consequently, the relative merits of IAF over established modality-balancing approaches remain insufficiently demonstrated.
Questions for authors
  1. Regarding the causal decomposition experiments in Section 4.3, how are the classifier parameters obtained for the frozen-classifier conditions? Additionally, which components, including the gating network and projection layers, are updated in each experimental condition?
  2. In the MUStARD dual-teacher setup, T1T_1 is defined as the primary teacher, yet w1w_1 varies depending on the dominant modality. Is this a notation inconsistency? How are the two teachers and their weights assigned in each configuration?
  3. The paper defines strong-modality collapse based on Accpath<AccuniAcc_{path}<Acc_{uni}. How do the authors distinguish pathway-level predictive degradation from the loss of task-relevant information? Moreover, since the dominant modality in IAF bypasses cross-modal attention, its CKA score of 1.0 in Table 9 appears to follow directly from the architecture. What additional insight does this result provide?
  4. Appendix E compares the proposed method with existing approaches on three datasets, excluding MultiHuSE due to differences in evaluation protocols. Given the lack of direct comparisons with gradient modulation, Pareto-based optimization, and prototype-guided methods under consistent experimental settings, what experimental evidence supports IAF's advantages over these existing modality-balancing approaches?
2 comments

We thank the reviewer for the careful reading and detailed feedback. Our responses follow:

W1 (Figure 2): We will redraw it as two separate stages, with the pure and attend pathways in separate lanes.

W2 (MAKD): In Stage 1, the pretrained encoders stay frozen, and we will specify the trainable components in Section 3.5 and Appendix O. For the dual-teacher setup on MUStARD, T1 is the audio teacher (Dasheng), and T2 is the video teacher (PE-Core) in every run. The weight w1 is the audio weight: 0.8 in the audio-primary setting and 0.2 in the video-primary setting. We will rewrite Eq. 8 with explicit audio and video weights so “primary” no longer depends on the notation. MAKD is not intended to improve every case. On Food-101, it lowers unimodal accuracy by 0.08 and 0.78 pp, and IAF goes from 94.72 to 94.27 on E5+PE-Core, as reported in Section 4.4. The two Food-101 modalities differ by only 0.78 pp, and every fusion method gains 5 to 9 pp, so there is little for MAKD to fix.

W3 (collapse versus information loss, CKA): Pathway accuracy measures predictive performance, not information content, and we will describe collapse as pathway-level predictive degradation. Table 4 gives partial evidence: with attention on and the classifier frozen (Cell 3), the dominant pathway stays within 1.4 pp of its ceiling on all four MultiHuSE configurations. For the IAF dominant pathway, CKA = 1.0 follows from the design, and we will mark it as a design check. The informative CKA values are the Symmetric ones, and Section 4.3 notes that CKA size does not explain accuracy loss (CKA 0.962 with a 42.6 pp drop).

W4 (configuration names): We will use full names in all tables, such as BERT+Dasheng+VideoMAE and GloVe+COVAREP+OpenFace.

W5 (Section 4.3 setup, UR-FUNNY): The classifier is initialized identically across the four cells (Section 3.2). We will state where the frozen weights come from and which components are trained in each cell. On UR-FUNNY, classifier drift alone causes the largest drop, so the interaction is not consistent across datasets. We report it as an exception in Section 4.3 and will not describe the interaction as general.

W6 (baselines): AGM is compared in Table 10, but not under our setting. PMR, MMPareto and the OGM-GE extension are cited and not run. We will add them on our own features where feasible, including on MultiHuSE.

Q1. See W5. Q2. See W2. Q3. See W3. Q4. See W6.

My concerns remain unresolved, so I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.