acceptodds
Under review as a conference paper at ICLR 2027

MRCF: Rethinking Modality Reliability in Multimodal Learning with Incomplete Observations

Abstract

Multimodal Sentiment Analysis (MSA) integrates text, audio, and vision to infer human affect, yet real-world multimodal observations are often incomplete. Existing methods for incomplete-observation MSA mainly follow two paradigms. Reconstruction-based methods recover missing information from observed modalities, while joint-representation methods learn directly from incomplete inputs. Although effective, these methods usually treat modality reliability only implicitly within representation learning or fusion design rather than modeling it explicitly. We argue that modality reliability is a central variable in incomplete-observation settings. Failure to model it explicitly gives rise to two related issues. The first is reliability mismatch, in which the affective evidence retained by each modality varies across samples and missing rates. The second is reliability propagation bias, in which messages from degraded modalities may adversely affect cross-modal interaction and predictive performance. To address these issues, we propose MRCF, a Modality Reliability-Calibrated Framework for MSA with incomplete observations. MRCF contains a Reliability-Aware Branch that estimates sample-specific modality reliability from intramodal quality cues and cross-modal semantic consistency, a Reliability-Guided Interaction Branch that uses the estimated scores to modulate cross-modal information flow, and a Reliability-Calibrated Fusion Module that integrates reliability and semantic cues for final prediction. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show that MRCF achieves strong performance under standard incomplete-observation protocols. Further analyses provide evidence that explicit reliability modeling helps mitigate reliability mismatch and reliability propagation bias during interaction and fusion.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Multimodal Learning, Multimodal Sentiment Analysis, Incomplete Observations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xNSGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MRCF for multimodal sentiment analysis under incomplete observations. It estimates sample-specific modality reliability using intramodal quality cues, cross-modal semantic consistency, and complete-observation prediction agreement. The estimated reliability guides cross-modal message passing and adaptive fusion, while a complete-observation anchor provides training-time calibration. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS report competitive results under intra-modal and fixed-modality missingness, supported by component ablations and reliability-related analyses.

Critical strengths
  1. The paper addresses a relevant problem by explicitly modeling sample-specific modality reliability under incomplete observations. Incorporating reliability into both source-side message weighting and target-side interaction updates is reasonably motivated and provides a coherent approach to reducing the influence of degraded modalities.

  2. The experimental evaluation covers multiple benchmarks and missingness protocols, with generally competitive results.

Critical weaknesses
  1. Section 2.3 constructs modality reliability targets from the agreement between incomplete unimodal predictions and the complete-observation prediction. However, agreement with the full-modality prediction does not necessarily indicate the quality or correctness of the underlying modality. In particular, a modality providing correct but conflicting evidence may receive a low reliability score when the complete-observation prediction is incorrect. Cross-modal semantic consistency may introduce a similar bias against informative minority modalities. Although Figure 3 reports positive correlations between estimated reliability and the prediction-agreement proxy, this diagnostic follows the same criterion used for training and therefore does not independently establish actual modality trustworthiness. I would like to see more targeted experiments, such as evaluating reliability against ground-truth prediction errors or independently assessed modality quality, and examining cases where the complete-observation prediction is incorrect.

  2. Several closely related approaches are absent, particularly EBMC (CVPR 2026) [1], which explicitly estimates instance-level modality trust and adaptively calibrates multimodal learning, and Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities (IJCAI 2026) [2], which uses complete-modality references and decision-level consistency to suppress unreliable information. These methods are particularly relevant to the central motivation and design of MRCF. Moreover, while the existing ablations demonstrate the benefits of reliability-guided interaction, the directional message-strength analysis largely reflects the behavior explicitly encouraged by the routing formulation, rather than independently establishing that erroneous information propagation is reduced. Comparisons under matched missingness protocols, together with more direct evidence connecting reliability-guided message suppression to prediction improvements, would strengthen the empirical conclusions.

Refs:

[1] Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis, Kang He et al., CVPR 2026.

[2] Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities, Chenglizhao Chen et al., IJCAI 2026.

[3] TMDC: A Two-Stage Modality Denoising and Complementation Framework for Multimodal Sentiment Analysis with Missing and Noisy Modalities, Yan Zhuang et al., AAAI 2026.

Questions for authors
  1. Please broaden the comparison with recent relevant methods, particularly EBMC (CVPR 2026) [1] and Controlling Decision Drift (IJCAI 2026) [2], under matched missingness and feature protocols where feasible. Clarifying the methodological differences would also help establish the contribution of MRCF.

  2. How does the reliability proxy in Section 2.3 distinguish genuinely reliable modality evidence from agreement with a potentially incorrect complete-observation prediction? Do the existing analyses examine cases where the complete-observation prediction disagrees with the ground-truth sentiment?

  3. Is there evidence that the estimated reliability scores correlate with actual unimodal prediction errors or independently assessed input quality, beyond their correlation with the prediction-agreement proxy used during training?

  4. Figure 4 shows stronger message routing from higher- to lower-reliability modalities, as encouraged by the proposed interaction formulation. Do the authors have evidence that this routing actually reduces harmful information propagation, rather than merely reproducing the designed weighting behavior?

  5. I would like to see more targeted experiments that evaluate the robustness of reliability-guided interaction when reliability estimates are imperfect, such as perturbing the estimated reliability scores, comparing alternative reliability targets, or analyzing samples with misleading cross-modal agreement.

The paper presents a reasonably motivated framework and reports competitive results with extensive supporting analyses. However, my main concerns are that the proposed reliability proxy has not been sufficiently validated against actual modality trustworthiness, and that the experimental comparisons do not adequately cover closely related recent methods. These issues currently limit confidence in the central reliability-modeling claims and the empirical advantages of MRCF. I would be willing to raise my score if the authors can adequately address the concerns above, particularly through stronger independent reliability validation and more comprehensive comparisons with relevant baselines.

Reviewer_SodAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the MRCF for MSA with partially missing observations or entirely unavailable text, audio, or vision modalities. Specifically, the Reliability-Aware Branch combines intramodal quality cues and cross-modal semantic consistency to estimate sample-specific modality reliability. The Reliability-Guided Interaction Branch adjusts cross-modal message weights and target-side update magnitudes to reduce interference from degraded evidence. The Reliability-Calibrated Fusion Module integrates semantic and reliability cues for prediction and uses complete-observation representations as training-time anchors for alignment. Experimental results on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate the effectiveness of the proposed MRCF under the evaluated missingness conditions.

Critical strengths
  1. The proposed MRCF framework incorporates sample-specific modality reliability into both cross-modal interaction and final fusion. The overall framework and design rationale are reasonable, and the method achieves competitive performance in MSA with incomplete observations.

  2. This paper evaluates intra-modal missingness on three datasets and fixed-modality missingness on two datasets, comparing MRCF with a range of existing methods. Ablation studies on the framework components, reliability cues, and training objectives further demonstrate the effectiveness of the individual designs.

Critical weaknesses
  1. Reliability supervision is constructed from the agreement between incomplete unimodal predictions and complete-observation predictions. However, prediction agreement does not necessarily imply prediction correctness. The current reliability analysis mainly verifies the consistency between the estimated scores and this proxy, without establishing an independent relationship with actual predictive quality. Furthermore, label supervision for the auxiliary prediction heads and gradient handling for the proxy targets are not sufficiently explained.

  2. This paper provides analyses of reliability correlations and cross-modal interaction directions. However, these results primarily demonstrate agreement with the proxy and the intended routing behavior, rather than directly verifying whether reliability-guided interaction reduces the negative influence of degraded modalities on other modalities and the final prediction.

Questions for authors
  1. Do the unimodal prediction heads and the complete-observation reference branch receive separate supervision from sentiment labels? When constructing the proxy reliability targets, are gradients stopped through the entire target?

  2. At a fixed missing rate, do the estimated reliability scores correlate with the unimodal prediction errors measured against ground-truth sentiment labels?

  3. Could the authors provide a controlled experiment that progressively corrupts one modality while keeping the others unchanged, showing how reliability scores and predictions change? A comparison with a model without reliability-based gating would help verify whether the proposed mechanism reduces interference from degraded evidence.

Reviewer_eGcEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

I appreciate the authors’ observation of reliability propagation bias. However, I believe many of the other motivations have already been explored in prior work, which is not sufficiently discussed. The presentation is also somewhat difficult to follow, with many formulas, dense text, and numerous symbols. In addition, I have the following concerns:

(1) The so-called reliability proxy is closer to “agreement with the complete multimodal prediction” than to modality reliability itself. Even an uncorrupted modality may differ substantially from the complete multimodal prediction because it contributes less to the task, provides complementary information, or conflicts with other modalities. Conversely, a corrupted modality may coincidentally agree with the complete prediction.

(2) There is a logical inconsistency in Reliability-Guided Interaction Branch Module. A less reliable target modality receives a larger update, but the cross-modal attention also relies more heavily on the query generated from this unreliable target. If the target modality is severely corrupted or even completely missing, its query is itself unreliable. The model then uses this low-quality query to determine what information to retrieve from other modalities while applying a larger update.

(3) The fusion module uses reliability estimated before interaction rather than after interaction. $\rho_i^{(m)} $ is estimated from the original $h_i^{(m)} $, after which Reliability-Guided Interaction Branch Module updates it to $\tilde{h}_i^{(m)} $. However, Reliability-Calibrated Fusion Module still uses the original $\rho_i^{(m)} $ to fuse $\tilde{h}_i^{(m)} $. If Reliability-Guided Interaction Branch Module is intended to correct a low-reliability modality, its reliability should theoretically change after correction. Reusing the old low reliability score may therefore continue to suppress a representation that has already been corrected.

(4) Minor issue: the main text states that the results in Table 4 are averaged over missing rates from 0.1 to 0.9. According to the appendix results, the range should be 0.0 to 0.9.

Critical strengths

The observation of reliability propagation bias is interesting. It indicates that evidence from the lowest-scoring modality may increasingly disrupt cross-modal interaction and fusion rather than provide useful support.

Critical weaknesses

(1) The paper claims that existing methods do not explicitly and systematically model modality reliability under incomplete observations. However, to my knowledge, there are already a number of related methods, such as DEAR (ACL 2026, “DEAR: Distributional Error-Aware Reliability for Robust Multimodal Sentiment Analysis with Missing Modalities”). In addition, many existing studies model uncertainty or reliability in this field, but the paper provides insufficient discussion of these related works.

(2) The proposed reliability metric lacks empirical validation or theoretical analysis, so it remains unclear whether it can accurately reflect modality reliability.

Questions for authors

Please refer to paper summary and weaknesses.

Comments on the paper

1 comment

Dear reviewers,

As the authors have posted their responses, I would encourage you to actively parcitipate in the dicussion.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.