- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies "strong-modality collapse," a phenomenon where symmetric cross-modal attention degrades the predictive accuracy of the dominant modality pathway during multimodal fusion. Using a causal decomposition, the authors attribute this degradation to the compounding interaction between cross-modal feature alteration and joint classifier adaptation. To address this, they propose a two-stage framework: Modality-Aware Knowledge Distillation (MAKD) to strengthen weaker encoders prior to fusion, and Inverted Asymmetric Fusion (IAF), which shields the dominant modality by bypassing cross-modal attention and freezing its unimodal classifier while allowing weaker modalities to attend to it. Experiments across four benchmarks (MultiHuSE, UPMC Food-101, UR-FUNNY, and MUStARD) validate the approach.
Critical strengths
- Pathway-level evaluation catches what fused metrics miss. 2.Empirical evaluation across four diverse datasets and multiple authentic pretrained backbones.
Critical weaknesses
- Preservation holds by construction. With frozen features, no attention and a frozen head, the pure pathway equals the unimodal model. Ceiling accuracy and CKA = 1.0 (Tables 6, 7, 9) are therefore guaranteed.
- The collapse may be benign. Under a summed-logit loss, heads co-adapt rather than act as standalone classifiers. On Food-101 BERT+PE-Core, the image pathway loses 42.6 pp, yet text rises to 85.38% and fused accuracy exceeds the ceiling. Symmetric Fusion falls below the ceiling only twice, both within one SD.
- Statistical support is thin. The interaction reverses on UR-FUNNY and vanishes for MUStARD E5+VideoMAE audio, and no confidence intervals are given. "Causal" overstates a five-fold ablation. No within-fold validation split is described for selecting hyperparameters or the dominant modality.
- Gains over simple baselines are weak. Without MAKD, IAF trails early or late fusion in five of eleven settings and Symmetric Fusion in three. On UR-FUNNY, most of the gain comes from MAKD (Symmetric+MAKD: 69.18%), and Δ uses the non-distilled ceiling (62.11% vs. 65.19%).
- Novelty is limited. Directional cross-modal attention [a] and language-guided asymmetric fusion [b] already exist. Fixing one branch while fitting the others resembles ReconBoost [c]. MAKD is well-studied crossmodal distillation (Hinton et al., 2015; [d]).
- SOTA comparisons are missing: OGM-GE [e], UMT/UME [f], MLA [g] and ReconBoost [c]. Tables 10 and 11 omit HKT [h] (77.36% UR-FUNNY, 79.41% MUStARD), and their baselines use older encoders.
[a] Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." ACL, 2019. [b] Zhang, Haoyu, et al. "Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis." EMNLP, 2023. [c] Hua, Cong, et al. "Reconboost: Boosting can achieve modality reconcilement." ICML, 2024. [d] Xue, Zihui, et al. "The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation." ICLR, 2023. [e] Peng, Xiaokang, et al. "Balanced multimodal learning via on-the-fly gradient modulation." CVPR, 2022. [f] Du, Chenzhuang, et al. "On uni-modal feature learning in supervised multi-modal learning." ICML, 2023. [g] Zhang, Xiaohui, et al. "Multimodal representation learning by alternating unimodal adaptation." CVPR, 2024. [h] Hasan, Md Kamrul, et al. "Humor knowledge enriched transformer for understanding multimodal humor." AAAI, 2021.
Questions for authors
- With fine-tuned encoders or token-level attention, does the dominant pathway still match its ceiling in accuracy and CKA?
- Does a linear probe re-fit on post-fusion dominant features recover the ceiling? Does the pathway drop correlate with fused accuracy across configurations?
- Please report confidence intervals for the interaction term per configuration, explain the UR-FUNNY reversal, and state how hyperparameters and the dominant modality were chosen within folds.
- Does IAF+MAKD beat Early/Late Fusion+MAKD with identical teachers, and how does Δ change against distilled ceilings?
- What is IAF's fused accuracy without CMA, i.e., a frozen dominant head plus a trained residual late-fusion branch?
We thank the reviewer for the detailed review. Our responses follow:
W1 (preservation by construction): Yes, this follows from the design (Section 3.3): with frozen features, no attention and a frozen head, the IAF dominant pathway is the unimodal model, so ceiling accuracy and CKA = 1.0 are expected. What does not follow from the design is the fused accuracy, where IAF+MAKD is above the unimodal ceiling in all 11 configurations (Table 5), and the live-encoder check (Appendix F, one fold), where the text pathway is 2.70 pp below its ceiling with a frozen head, versus 9.54 pp for Symmetric Fusion.
W2 (collapse may be benign): Symmetric Fusion is below the unimodal ceiling in 2 of 11 configurations, both within one SD (Table 2). On Food-101 BERT+PE-Core, the image pathway falls from 85.46 to 42.89 while the text pathway rises from 76.08 to 85.38, and fused accuracy is 90.68. We will reword the claim as a pathway-level result that does not always lower fused accuracy. The pathway still matters when modalities are missing: in Appendix G, IAF matches or beats Symmetric Fusion in all 12 MultiHuSE comparisons where at least one non-dominant modality is present.
W3 (statistical support): The interaction term (Eq. 4) is above 3 pp in 10 of 15 pathway rows. It is near zero or negative in five: four MUStARD rows and UR-FUNNY, which we report as an exception in Section 4.3. “Most MUStARD configurations” should read “half”. We will replace “causal” with “factorial ablation”, add confidence intervals, and describe the validation split used to choose hyperparameters and the dominant modality.
W4 (gains over simple baselines): IAF is evaluated together with MAKD, so the fair comparison is IAF+MAKD against Symmetric+MAKD with the same teachers, where IAF is higher in 10 of 11 configurations (Table 5). On UR-FUNNY, MAKD accounts for most of the gain (Symmetric+MAKD 69.18, IAF+MAKD 69.80). Measured against the distilled dominant model, IAF+MAKD still gains +2.95 and +4.53 on MultiHuSE with BERT and +4.61 on UR-FUNNY. Elsewhere, the dominant model is the teacher, so the ceiling is unchanged.
W5 (novelty): MulT [a] applies directional cross-modal attention to every target modality, so no modality is shielded. IAF shields one modality, chosen from unimodal accuracy for each dataset (audio and video on MUStARD, where text is not dominant), and freezes its classifier, which the 2x2 ablation identifies as the second factor in the collapse. ReconBoost [c] alternates training so that one modality is updated while the others stay fixed, while IAF trains all pathways jointly with a structural shield. We will cite and discuss [b] and [c]. MAKD uses standard distillation losses (Eq. 7); it differs in placement before fusion and in using the dominant modality as teacher (Appendix M).
W6 (comparisons): OGM-GE, UMT/UME, MLA and ReconBoost were not run, and we will add them on our features where feasible. We will add HKT to Tables 10 and 11: it reports 77.36 on UR-FUNNY (IAF 69.80) and 79.41 on MUStARD (IAF 83.33). HKT fine-tunes ALBERT and uses external humor knowledge, which our setup does not. On UR-FUNNY we use the benchmark’s original features, for consistency with prior work. On MUStARD we use newer frozen encoders than the published baselines, so that comparison is indicative only, and we will state this in Appendix E.
Q1: With a fine-tuned encoder, the dominant pathway does not exactly match its ceiling. In the one fold we tested (Appendix F, LoRA on the text encoder), the text pathway was 2.70 pp below its ceiling with IAF’s frozen head, versus 9.54 pp for Symmetric Fusion. CKA was not computed there, and token-level attention was not tested. Both will be listed as limitations.
Q2: We did not run the linear-probe refit, and we will add it. On the correlation, a Spearman test computed from Tables 2 and 3 shows that the drop in the dominant pathway does not track fused accuracy across the 11 configurations (rho = -0.22, p = 0.52). For example, on Food-101 BERT+PE-Core, the image pathway falls by 42.6 pp while fused accuracy is 5.2 pp above the ceiling.
Q3: We will add confidence intervals for the interaction term in every configuration. On UR-FUNNY, classifier drift alone gives 48.31, which is chance for this binary task, and attention raises it to 57.83. The dominant modality is the one with the highest standalone validation accuracy (Eq. 1), and hyperparameters were tuned manually on validation performance (Appendix O). We will describe the validation split within each fold.
Q4: Early and Late Fusion with MAKD were not run, and we will add them with identical teachers. The only same-teacher comparison in the paper is with Symmetric Fusion (Table 5). Against the distilled dominant model, IAF+MAKD gains +2.95 and +4.53 on MultiHuSE with BERT and +4.61 on UR-FUNNY, versus -0.79, +1.79 and +3.99 for Symmetric+MAKD.
Q5. Not run, and we will add it.