Consensus-Complement Subspace Fusion for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) seeks to infer the sentiment of an utterance by integrating its language, acoustic and visual streams, whose importance and agreement vary across utterances. Traditional fusion methods either align the modalities to exploit their agreement or decouple them into invariant and specific parts, with heuristic losses whose balance is tuned by hand. Recent advances have placed alignment on a principled footing by driving the representation matrix of an instance towards rank one. However, full alignment discards modality-specific information, and no rule states how much of it should be retained. To address the challenges, in this paper, we propose PRISM, a novel framework that fuses modalities through an exact orthogonal decomposition of the instance matrix. Specifically, we are grounded in a theoretical insight. The rank-one truncation of the instance matrix spans a consensus subspace, and its Eckart-Young residual carries modality-specific innovation, termed the complement. We prove that full alignment incurs an excess risk equal to the label variance explained by modality-specific latents. Under a linear-Gaussian latent model, the Bayes-optimal predictor instead adds a consensus score to individually shrunk complement scores. PRISM realises this predictor. It weights the consensus by the leading right singular vector, a training-free modality router certified by the consensus ratio. It shrinks each complement by a certificate-driven gate, the minimum-mean-square-error coefficient of the specific score. Besides, a Davis-Kahan analysis yields a spectral-gap regulariser that stabilises the decomposition. Extensive experiments on CMU-MOSI, CMU-MOSEI and CH-SIMS benchmarks demonstrate that PRISM achieves state-of-the-art performance.