Strong-Modality Collapse in Multimodal Fusion: Causal Analysis and Structural Solution

Multimodal fusion is expected to benefit from combining complementary signals. However, causal pathway isolation shows that standard symmetric cross-attention can degrade the predictive performance of the dominant modality, in some cases pushing the fusion model below its unimodal baseline. We call this failure strong-modality collapse. We find that it results from the interaction of two factors: cross-modal attention alters the dominant representation, while joint optimization adapts its classifier to these changes. In most configurations, neither factor alone causes substantial degradation, but together they produce losses of up to 42.6 percentage points. This finding motivates Inverted Asymmetric Fusion (IAF), which shields the dominant modality from cross-modal attention while allowing weaker modalities to attend to it. Because structural shielding alone cannot address initial feature imbalances, we introduce Modality-Aware Knowledge Distillation (MAKD) to strengthen weaker encoders before fusion. Across four multimodal benchmarks, IAF preserves the dominant pathway at its unimodal ceiling and improves multimodal performance over the strongest unimodal baseline.