ContriFuse: Contribution-Aware Fusion of Auxiliary Descriptions for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) integrates textual, acoustic, and visual signals to infer sentiment polarity and intensity. Existing methods typically encode inputs from different modalities into feature representations and perform sentiment prediction through cross-modal fusion. Recent studies further introduce auxiliary information, such as descriptions of audio-visual content and contextual knowledge, to enrich the semantic information contained in modality representations. Existing approaches mainly use such auxiliary textual information to augment or align modality representations, followed by joint, gated, or routed fusion for sentiment prediction. However, auxiliary descriptions from different modalities do not contribute equally to sentiment prediction; their usefulness depends not only on the corresponding original modality information, but also on auxiliary descriptions from other modalities. To address this issue, we propose ContriFuse, a multimodal sentiment analysis framework that adaptively regulates the contribution of each modality’s auxiliary descriptions before integrating them with the original modality features. First, ContriFuse constructs an auxiliary description representation for each modality from descriptions generated by multiple large language models. The resulting auxiliary representations are then mapped into shared and modality-specific spaces, allowing subsequent regulation to consider both information shared across modalities and information specific to each modality. ContriFuse further combines the original modality features, auxiliary description representations, and semantic similarities among descriptions from different modalities to determine the extent to which each modality’s auxiliary description should contribute to the current prediction. Finally, ContriFuse fuses the modulated auxiliary description representations with the corresponding modality features to predict sentiment polarity and intensity. Experiments on four widely used English and Chinese sentiment analysis benchmarks evaluate ContriFuse using both regression and classification metrics and demonstrate its effectiveness.