acceptodds
Under review as a conference paper at ICLR 2027

Consensus-Complement Subspace Fusion for Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis (MSA) seeks to infer the sentiment of an utterance by integrating its language, acoustic and visual streams, whose importance and agreement vary across utterances. Traditional fusion methods either align the modalities to exploit their agreement or decouple them into invariant and specific parts, with heuristic losses whose balance is tuned by hand. Recent advances have placed alignment on a principled footing by driving the representation matrix of an instance towards rank one. However, full alignment discards modality-specific information, and no rule states how much of it should be retained. To address the challenges, in this paper, we propose PRISM, a novel framework that fuses modalities through an exact orthogonal decomposition of the instance matrix. Specifically, we are grounded in a theoretical insight. The rank-one truncation of the instance matrix spans a consensus subspace, and its Eckart-Young residual carries modality-specific innovation, termed the complement. We prove that full alignment incurs an excess risk equal to the label variance explained by modality-specific latents. Under a linear-Gaussian latent model, the Bayes-optimal predictor instead adds a consensus score to individually shrunk complement scores. PRISM realises this predictor. It weights the consensus by the leading right singular vector, a training-free modality router certified by the consensus ratio. It shrinks each complement by a certificate-driven gate, the minimum-mean-square-error coefficient of the specific score. Besides, a Davis-Kahan analysis yields a spectral-gap regulariser that stabilises the decomposition. Extensive experiments on CMU-MOSI, CMU-MOSEI and CH-SIMS benchmarks demonstrate that PRISM achieves state-of-the-art performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Multimodal Sentiment Analysis, Subspace Learning, Modality Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_RWZ9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes PRISM, a consensus–complement subspace fusion framework for multimodal sentiment analysis. PRISM employs singular value decomposition (SVD) to decompose the representation matrix of each multimodal instance into a leading consensus subspace and its orthogonal modality-specific complement subspaces. The leading right singular vector provides instance-level consensus fusion weights, while the learned certificate gates adaptively control the contributions of modality-specific complement scores. It analyzes the information loss caused by full alignment, the shrinkage mechanism of complement components under a linear-Gaussian model, and the stability of the subspace decomposition through a spectral-gap analysis. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS, including ablation studies and robustness analyses, demonstrate the effectiveness of the proposed method.

Critical strengths

None

Critical weaknesses

This paper presents a promising geometric perspective on multimodal fusion, supported by extensive experimental comparisons and substantial theoretical analysis. However, the current version somewhat overstates both the scope of the proposed full-alignment impossibility result and the Bayes-optimal interpretation of the learned gating mechanism. In addition, the central trade-off between modality alignment and modality-specific information still depends on a regularization weight tuned on the validation set. Moreover, the paper lacks sufficiently up-to-date and comprehensive comparisons with directly relevant works published in 2026. Some of the experimental comparisons also appear not to be entirely fair or accurate, particularly the comparison with GRAM.

Questions for authors
  1. Scope of Theorem 4. Does the predictive incompleteness of full alignment apply only to independently encoded modalities, or also to representations that have undergone cross-modal interaction?
  2. Gap between theory and implementation. The Bayes-optimal gate is derived under an oracle linear-Gaussian setting, whereas PRISM uses learned gates and a validation-tuned wgw_g. Could the authors explain in more detail which components of the implemented method have theoretical optimality guarantees, which are empirical design choices, and the rationale behind these choices?
  3. Comparison with recent work. Why are closely related 2026 MSA methods not included in the experiments? For example, CACR also considers shared/specific representations and cross-modal conflict.
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer RWZ9,

We hope this message finds you well. We are writing to kindly follow up on our rebuttal to check whether our responses have fully addressed your concerns. If so, we would greatly appreciate it if you would consider raising the final score. If you have any further concerns, please don't hesitate to discuss with us. We greatly value your feedback and will continue improving the final version based on the discussions.

Thank you again for your time and thoughtful reviews.

Best regards,

Authors

Reviewer_RWZ9OpenReview ↗

Official Comment by Reviewer RWZ9

Thanks for authors addressed most of my concerns. I will raise up my score.

AuthorsOpenReview ↗

Follow-up on Remaining Concerns

Dear Reviewer RWZ9,

Thank you very much for your positive feedback and for taking the time to consider our rebuttal. We sincerely appreciate your acknowledgment of our efforts to address your concerns.

We noticed that you mentioned most of your concerns have been addressed. We would be grateful if you could let us know whether there are any remaining concerns that might affect your overall assessment of the paper. We would be happy to provide further clarifications or additional evidence for the final positive score.

Thank you again for your constructive feedback and valuable time.

Best regards,

Authors

Reviewer_2DVmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRISM for multimodal sentiment analysis. It decomposes each instance’s modality representation matrix into a rank-one consensus component and orthogonal complements. The final prediction combines a consensus score with learned, instance-dependent gates applied to the complement scores. Training additionally uses unimodal and component-level supervision, a spectral-gap regularizer, and a penalty on negative cross-modal inner products. The theoretical analysis characterizes the information loss induced by exact full alignment, derives an additive Bayes predictor and optimal shrinkage coefficients under a linear-Gaussian latent model, and bounds the sensitivity of the consensus direction to perturbations. Experiments cover CMU-MOSI, CMU-MOSEI, and CH-SIMS, including baseline reruns, component ablations, spectral diagnostics, and robustness evaluations. PRISM improves over the rerun baselines, with particularly notable gains on CH-SIMS.

Critical strengths

The paper presents a clear and computationally compact fusion formulation. The instance-wise orthogonal decomposition provides an explicit way to separate cross-modal agreement from residual information using only a small Gram-matrix eigenproblem. The theoretical analysis offers useful conditional explanations for why exact full alignment can discard predictive information and why the spectral gap matters for the stability of the decomposition.

The empirical evaluation is substantial. It includes three datasets, repeated training runs, extensive component ablations, spectral diagnostics, and comparisons with learned constant gates, a learned softmax router, uniform consensus weights, and alternative alignment objectives. Appendix C.9 and the aligned-baseline reruns in Appendix C.2 also help address the MOSI feature-alignment mismatch. The reported improvements over rerun EMOE on CH-SIMS, namely 3.2 percentage points in Acc-2, 4.2 points in Acc-3, and 0.024 lower MAE, constitute meaningful empirical evidence for the method.

The paper also does a comparatively good job of stating several limitations of its theoretical interpretation. In particular, it distinguishes scalar-score shrinkage from vector editing, notes the restriction associated with nonlinear prediction heads, and acknowledges that only the instance-dependent part of the gate can affect risk when the prediction head is freely learned.

Critical weaknesses

The statistical significance results are inconsistent with the reported sample sizes. Tables 1 and 4 state that PRISM uses ten MOSI seeds and the baselines five. Under these sample sizes, the reported Acc-7 results for PRISM and Self-MM yield a two-sided Welch test of approximately p=0.0504, rather than p<0.01, while their MAE results yield approximately p=0.119, rather than p<0.05. In contrast, Table 6 states that ten seeds were used on MOSI, and its annotations are consistent with using ten runs for both methods. The authors should clarify which runs were used and provide per-seed results or statistical-analysis outputs.

The connection between the theoretical MMSE shrinkage rule and the learned certificate gates is incomplete. Theorem 5 derives optimal coefficients under a linear-Gaussian model with an oracle shared direction and specified SNRs, whereas the implemented gates are produced by a neural network trained through the task objective. Although Equation (5) correctly gates scalar complement scores and the ablations support the gates' practical utility, no result shows that the learned outputs estimate inverse SNR or inherit the theorem's MMSE optimality. Claims describing the implemented gates as MMSE coefficients should therefore be qualified.

Questions for authors

Statistical clarification. Which exact runs and sample sizes were used for the MOSI comparisons in Table 6? Does Table 6 use ten baseline runs that are not included or described in Tables 1 and 4? Please reconcile the sample-size statements, specify whether the Welch tests were one-sided or two-sided, and provide the corresponding per-seed results or analysis outputs. It would also be helpful to clarify which set of runs generated the means and standard deviations in Tables 1 and 4.

Scope of the performance claim. Is the state-of-the-art claim restricted to rerun implementations under the authors’ evaluation protocol? Table 1 contains published results that outperform PRISM on several metrics, so an unrestricted SOTA claim is not established. Please also disclose the validation and hyperparameter-search budgets used for PRISM and the strongest rerun baselines, beyond the stated use of released baseline hyperparameters and validation-MAE model selection.

Scope of the GRAM comparison. Section 4.3 states that the evaluated GRAM-style objective removes the original contrastive normalization. Does the reported collapse support a conclusion only about this modified, unconstrained per-instance penalty, or is it intended as evidence about the original GRAM method? The wording of the experimental discussion should preserve this distinction.

Reviewer_JPNqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the tension between modality alignment and the preservation of modality-specific information in multimodal sentiment analysis. It theoretically characterizes the predictive cost of full alignment and motivates PRISM, a framework that uses spectral decomposition to separate representations into consensus and complement components by using linear-Gaussian latent model. Experiments on MOSI, MOSEI, and SIMS demonstrate its effectiveness, supported by component ablations, spectral analyses, parameter sensitivity studies, and robustness evaluations.

Critical strengths
  • The method is theoretically motivated, well designed, and interpretable. The consensus–complement formulation provides a clear perspective on multimodal fusion. The theoretical analysis under the stated assumptions informs several architectural choices, including additive prediction, certificate gating, and spectral-gap regularization.
  • The empirical evaluation is extensive and convincing. The paper compares against numerous previous methods and evaluates performance across multiple random seeds, with statistical analyses supporting the stability of the improvements. The broad set of ablations and robustness studies further strengthens the evidence for the proposed framework.
Critical weaknesses
  • The experimental evidence could be organized more clearly around the theoretically predicted sources of improvement. Although extensive, the evidence is somewhat scattered across the main text and appendix, making it difficult to trace how the latent-variable analysis translates into practical gains. A clearer validation chain could address three questions: whether consensus fusion produces the predicted balance gain, whether complements provide specialisation gains beyond the consensus, and whether certificate gating helps by suppressing unreliable complement contributions. The existing ablations establish the average utility of these components, but their connection to the theoretical mechanisms remains indirect. The authors could better integrate the existing evaluations and sample-level geometric analyses, examining how reductions in squared prediction error relate to the consensus ratio, innovation energies, loadings, and certificate gates. This would make the empirical support for the theory-guided design more explicit.

  • The explanation for the larger gains on SIMS requires more direct support. The paper attributes these gains to SIMS annotations being less language-dominated. While the existing results provide some support for this interpretation, the authors could substantiate it more directly. For example, extending Figure 8(b) to include the visual and acoustic modalities alongside language would enable a more complete comparison of the relationships between innovation energy and gating across modalities and datasets.

Questions for authors

Could the authors connect the existing evaluations with the sample-level geometric analyses? Specifically, when does consensus fusion improve over individual modalities, when do complements improve over the consensus prediction, and when does gating improve over the ungated alternative? Relating these improvements to the consensus ratio, innovation energies, and loadings would more directly assess the balance, specialisation, and shrinkage mechanisms motivating PRISM.

Reviewer_7rVrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PRISM, a multimodal sentiment analysis framework that decomposes modality representations into a shared consensus subspace and modality-specific complements using SVD. The consensus is fused with spectral-based weights, while the complements are adaptively retained through learned shrinkage gates. The authors provide theoretical analyses of the information loss caused by full alignment, optimal complement shrinkage under a linear-Gaussian model, and decomposition stability. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show improvements over several existing methods, supported by ablation studies and spectral analyses.

Critical strengths
  1. The consensus-complement decomposition provides a well-motivated way to preserve modality-specific information while exploiting cross-modal agreement. The theoretical analysis offers useful insights into why full alignment can be suboptimal.

  2. The experimental evaluation is relatively thorough. The authors reproduce strong baselines under a consistent protocol, report results across multiple seeds, and include extensive ablations. The improvements on CH-SIMS are particularly convincing.

Critical weaknesses
  1. The connection between the theoretical derivation and the actual implementation is not fully established. The optimal shrinkage coefficients are derived under a restrictive linear-Gaussian model, whereas the proposed gates are learned from spectral statistics and head predictions. It remains unclear whether these learned gates approximate the theoretically optimal coefficients.

  2. The performance improvements are relatively modest on some benchmarks, particularly CMU-MOSEI. Moreover, the learned router achieves nearly identical performance in Table 3, making the practical advantage of the proposed spectral routing less compelling.

Questions for authors
  1. Is there any empirical evidence that the learned certificate gates correlate with the actual reliability or signal-to-noise ratio of each modality? This would help validate the connection to the theoretical derivation.

  2. Since the learned softmax router performs almost identically to the SVD-based router in Table 3, could the authors clarify the practical advantages of the proposed routing mechanism, particularly in terms of computational cost?

Reviewer_436eOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method for multimodal sentiment analysis and provides relatively systematic theoretical analyses. However, the authors have not sufficiently explained the necessity of these theoretical analyses or their connection to the actual research problem. Furthermore, the experimental comparisons are relatively limited, with only one baseline method published in 2025, while the remaining methods are outdated.

The authors addressed most of my concerns. I raised up my score.

Critical strengths
  1. The paper is well-structured overall and proposes a new method for sentiment analysis.

2.The theoretical analysis is relatively strong and offers a certain degree of interpretability.

Critical weaknesses
  1. The authors claim that disentanglement and routing methods rely on manually adjusted loss weights. However, many methods already learn dynamic modality weights based on the features of each sample. Similarly, the authors claim that disentanglement and routing methods preserve specific information through auxiliary losses, while manually adjusted loss weights determine the balance between consistency and discrepancy. Such a characterization is also insufficiently rigorous, as it incorrectly generalizes the characteristics of some disentanglement methods to the entire research paradigm of disentanglement and dynamic routing.

  2. The authors state that “However, complete alignment leads to the loss of modality-specific information, and there is currently no rule for determining how much of such information should be preserved.” However, in deep learning, how much modality-specific information is preserved is generally learned adaptively by the network through training data and optimization objectives, and does not necessarily require a strict “rule.” Reasonable modality fusion does not necessarily require an explicit mathematical rule to determine the proportion of information to be preserved.

  3. In Equation (6), the authors still use manually specified hyperparameters to determine the overall balance between consistency and discrepancy. However, in the Introduction, one of the main issues criticized by the authors is precisely manual weight adjustment. This indicates that PRISM itself also suffers from this issue.

  4. The Method section is overly technical and very difficult to understand. Its connection to the motivation is insufficiently clear, and the overall coherence is also weak, resulting in poor comprehensibility.

  5. The experimental comparisons include too few SOTA methods from 2025 and 2026. Currently, only one method is from 2025, while all the others are from 2023 or earlier, making it difficult to effectively support the SOTA claim.

  6. Similarly, in the Introduction, the authors argue that existing methods lack explicit rules for determining how much modality-specific information should be preserved. However, in the experimental section, the authors still adaptively learn the balance between shared and specific information through training objectives and hyperparameter selection on the validation set.

Questions for authors

1.The authors claim that existing disentanglement and dynamic routing methods rely on manually adjusted loss weights to balance consistency and discrepancy. However, PRISM also uses manually specified hyperparameters. In this case, what is the fundamental difference between PRISM and the methods criticized by the authors?

2.The authors seek to use explicit rules to determine how much modality-specific information should be preserved. However, is this necessary? Alternatively, could the authors provide supporting evidence demonstrating that existing adaptive learning approaches have certain specific shortcomings?

3.Only one method included in the comparative experiments is from 2025, while all the others are from 2023 or earlier. This makes it difficult to effectively demonstrate the advancement of the proposed method. Could the authors compare their method with several methods published in 2026 to demonstrate its superiority?

2 comments

The authors addressed most of my concerns. I raised up my score.

AuthorsOpenReview ↗

Thanks for positive response

Thank you very much for your positive response and for taking the time to review our rebuttal! We are pleased to hear that our rebuttal have addressed your concerns. We sincerely appreciate your recognition of our work and the improved evaluation score. We will revise our paper in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks for positive feedback

We are grateful for your constructive feedback and careful consideration of our rebuttal. It is encouraging to know that most of your concerns have been resolved. These additional discussions will be included in the final version.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.