acceptodds
Withdrawn from ICLR 2027

High Uncertainty Is Not Always Harmful: Consistency-Guided Cross-Modal Uncertainty Rectification for Robust Multimodal Affective Computing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Affective Computing (MAC) has attracted increasing attention in recent years, aiming to predict human affective states by exploiting complementary signals across different modalities. Despite substantial progress, existing methods remain vulnerable to noisy inputs, missing modalities, and the intrinsic uncertainty of multimodal data, limiting their reliability and robustness. In this paper, we propose a Consistency-guided Cross-modal Uncertainty Rectification (CUR) framework. Unlike existing approaches that primarily adjust the fusion contribution of each modality according to its uncertainty, CUR leverages cross-modal consistency to rectify unreliable yet task-informative modalities, thereby enabling robust prediction. Specifically, we first probabilistically model modality-specific representations and jointly estimate their uncertainty and task utility. We then adaptively rectify the distribution of each target modality by aggregating reliable evidence from the remaining modalities. Finally, we introduce a dual-branch evidence fusion mechanism that integrates the original modality information with the rectified cross-modal consensus, suppressing unreliable information while preserving task-relevant cues. Extensive experiments across four MAC tasks and diverse degradation scenarios demonstrate the effectiveness and robustness of CUR.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Affective Computing, Uncertainty Modeling, Cross-modal Consistency, Uncertainty Rectification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QrrEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenges of unreliable and uncertain modalities in multimodal affective computing. To address these issues, the authors propose C²UR, a consistency-guided cross-modal uncertainty rectification framework. Specifically, it separately models modality uncertainty and task utility, employs reliability-weighted cross-modal consensus to rectify uncertain representations, and introduces dual-branch evidence fusion to preserve task-relevant information. Experiments across four affective computing tasks demonstrate improved prediction performance and robustness under noisy and missing modality conditions.

Critical strengths
  1. Meaningful research problem. The paper challenges the conventional assumption that highly uncertain modalities should always be suppressed, providing an interesting perspective on robust multimodal learning.

  2. Comprehensive experimental evaluation. Experiments across four affective computing tasks, together with ablation studies and robustness analyses, demonstrate the effectiveness of the proposed framework.

  3. The experiments consider different combinations of noise intensity and missing modality rates, providing evidence of the model's robustness under degraded inputs.

Critical weaknesses
  1. Probabilistic representation learning, Product-of-Experts, and uncertainty-aware fusion have been extensively studied. The authors should further clarify the unique contributions of their integration beyond existing multimodal uncertainty learning methods.

  2. The framework estimates modality uncertainty using the variance of learned Gaussian representations. However, it remains unclear whether this variance reliably reflects actual modality uncertainty. Further calibration analysis would strengthen this assumption.

  3. In Eq. (6), modality removal is simulated by replacing its representation with a zero vector. However, this operation may introduce out-of-distribution representations.

  4. The authors should compare training time, inference latency, and memory consumption with existing methods to evaluate the computational efficiency of the proposed framework.

Questions for authors
  1. Although the proposed framework separately estimates modality uncertainty and task utility, it remains unclear whether these two quantities are sufficiently independent. The authors should analyze their correlations under different noise levels and investigate whether the learned utility provides additional information beyond uncertainty.

  2. Stronger rectification of high-utility modalities may distort their task-relevant information. The authors should investigate how rectification strength affects the preservation of modality-specific features.

Reviewer_2WtFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Consistency-guided Cross-modal Uncertainty Rectification (C²UR) for multimodal affective computing. Its central motivation is that representation uncertainty and conditional task utility are distinct: an uncertain modality may still provide useful complementary information.

C²UR learns Gaussian modality representations and estimates modality utility using supervision derived from the task-loss increase caused by modality removal. It constructs reliability-weighted cross-modal Gaussian Product-of-Experts consensus distributions and uses predicted utility, target uncertainty, and auxiliary prediction consistency to control posterior rectification. A dual-branch fusion module combines utility-weighted original representations with rectified representations.

The paper evaluates five datasets spanning sentiment analysis, humor detection, sarcasm detection, and emotion recognition. It also reports a joint feature-noise and modality-missingness evaluation, component ablations, hyperparameter sensitivity, five-seed results, and computational comparisons.

Critical strengths

The distinction between modality utility and representation uncertainty is well motivated and relevant to multimodal fusion. The method responds coherently to this distinction by learning removal-based utility and preserving original evidence alongside rectified consensus. The ablations comparing learned utility with reliability-based weighting, and removing the original-evidence branch, provide useful evidence for these design choices.

The empirical evaluation is relatively broad and includes meaningful comparison controls. The paper covers five datasets and four affective tasks. Appendix B.4 states that several principal baselines use the same feature extraction pipeline and receive the same number of hyperparameter-search trials under the same validation criterion. The fixed-checkpoint joint-degradation evaluation, detailed ablations, five-seed statistics, and complexity analysis strengthen the empirical contribution.

Critical weaknesses

The training and validity of the removal-based utility teacher are insufficiently specified. Equations (5)–(7) derive utility targets using an auxiliary concatenation-and-MLP fusion network F_theta, whereas Equation (13) uses the final dual-branch architecture. Equation (14) does not explicitly include a task objective for the auxiliary fusion network, and the utility targets are stop-gradient. As written, it is unclear how F_theta is trained, whether it shares parameters with the final predictor, or whether a separate training stage or objective is omitted. This matters because the utility supervision is meaningful only if the teacher’s predictions and removal responses are meaningful. The paper also does not directly validate predicted utilities against measured removal effects or establish their agreement with contributions to the final predictor.

The evidence does not yet adequately connect posterior variance reduction to improved predictive reliability. The Gaussian Product-of-Experts covariance derivation in Appendix A.3 appears correct, but covariance contraction follows from adding positive precision and can occur even when auxiliary evidence shifts the mean in the wrong direction. Auxiliary agreement does not exclude jointly incorrect or correlated evidence. Figure 4 establishes aggregate sensitivity to missingness, while Figure 5 illustrates selected high-uncertainty, high-utility samples; neither establishes population-level reliability improvement. This concern is particularly relevant under missingness because zero-filled inputs still appear to be processed as Gaussian experts, without an explicit availability mask in the consensus equations. Broader paired error analyses and failure-case evaluations are needed to substantiate the central rectification interpretation.

Questions for authors

Clarification: How is the auxiliary fusion network F_theta in Equations (5)–(6) trained? Please specify its objective, parameter sharing, training schedule, and gradient paths. If an additional task loss or pretraining stage is used, please provide the complete training objective.

Assessment-changing validation: How accurately does the utility predictor recover measured removal-based utilities, both for the auxiliary teacher and for the final dual-branch predictor? Please report utility-ranking or correlation results under clean and degraded inputs. An oracle-utility diagnostic would also help establish whether utility-estimation error limits performance.

Clarification and assessment-changing validation: Are unavailable modalities excluded from the expert product and auxiliary-consistency calculation? If they remain as zero-input experts, what prevents two missing modalities from producing mutually consistent but misleading evidence? Please report results separately for the six nonempty incomplete-modality patterns.

Assessment-changing validation: Across the full test set, how often does rectification reduce prediction error, and how often does it increase error? Please provide paired before/after results stratified by utility and uncertainty, including cases where auxiliary modalities agree but are incorrect.

Assessment-changing validation: Can the strongest baselines be evaluated over matched random seeds and degradation realizations, with uncertainty estimates for the performance differences? The five-seed results for C²UR are useful, but do not establish the stability of the smaller margins over competing methods.

Reviewer_kK1gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

C2UR is a robust multimodal affective computing framework for noisy and missing modalities, motivated by the distinction between representation uncertainty and task utility—uncertain modalities may still carry useful task-relevant information. It combines utility–uncertainty modeling (Gaussian representations with posterior-variance uncertainty and a removal-based utility predictor), consistency-guided rectification (reliability-weighted cross-modal consensus via Product-of-Experts, adaptively fused into each modality's posterior), and dual-branch evidence fusion (utility-weighted original plus reliability-aware rectified representations). Evaluated on four affective computing tasks (CMU-MOSI, CMU-MOSEI, UR-FUNNY, MUStARD, CHERMA), it reports baseline improvements, robustness to joint noise and missingness, and ablation analyses. Its main contribution is explicitly separating modality utility from representation uncertainty and using cross-modal agreement to guide probabilistic refinement; however, whether these measures reliably identify useful cross-modal evidence, rather than merely adding modeling capacity and fusion, requires further validation.

Critical strengths
  1. The paper usefully separates task utility from representation uncertainty via leave-one-modality-out utility supervision, avoiding the assumption that uncertainty and usefulness are inversely related.

  2. Four-task evaluations with component-level ablations and joint noise–missingness experiments offer preliminary evidence that the components are complementary.

Critical weaknesses
  1. The central uncertainty-rectification mechanism is not sufficiently validated as a reliability mechanism, since neither posterior variance nor prediction agreement necessarily measures evidence correctness, and auxiliary modalities can agree on errors while Product-of-Experts still increases confidence.

  2. The claimed theoretical guarantee and observed uncertainty reduction are insufficient, as reduced covariance is directly induced by the update rule; direct validation is needed that uncertainty predicts reliability, consistency identifies trustworthy evidence, and rectification improves accuracy or calibration under controlled corruption.

  3. Robustness evidence remains limited because the continuous reliability-spectrum experiment relies on synthetic Gaussian noise and random masking, and joint-degradation analysis covers only sentiment rather than all four tasks.

  4. The gains are modest in several settings, and the contributions of uncertainty and consistency mechanisms are not sufficiently isolated from added representation and fusion capacity; stronger matched-capacity comparisons, repeated-run statistics, and tests under misleading or correlated auxiliary evidence are needed.

Questions for authors
  1. Does the uncertainty score quantitatively track actual modality degradation and prediction error, and how does it compare with calibrated or empirically estimated modality-quality scores?

  2. Can consistency-guided rectification avoid negative transfer when auxiliary modalities agree but are incorrect, and how does C2UR compare with disabling rectification or using uncertainty-based fusion alone?

  3. How much of the gain comes from the proposed mechanisms rather than added capacity or optimization, and can parameter- and compute-matched comparisons with simpler fusion or gating isolate their contribution?

  4. Are the improvements statistically reliable across multiple runs with significance tests, and do the reliability-spectrum conclusions generalize beyond sentiment analysis and synthetic Gaussian noise with random masking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.