acceptodds
Withdrawn from ICLR 2027

Modality-Aware Contrastive and Uncertainty-Regularized Emotion Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Emotion Recognition (MER) has attracted growing attention with the rapid advancement of human–computer interaction. However, different modalities exhibit substantial discrepancies in semantics, quality, and availability, leading to highly heterogeneous modality combinations and posing significant challenges to achieving consistent and reliable emotion understanding. To address this challenge, we propose the Modality-Aware Contrastive and Uncertainty-Regularized (MCUR) framework, which approaches MER from the perspective of representation consistency, aiming to enable robust emotion prediction across heterogeneous modality combinations. MCUR incorporates two core components: (1) Modality Combination-Based and Category-Based Contrastive Learning mechanism (MCB-CL), which encourages samples with the same emotion category and the same available modalities to be close in the representation space; and (2) Sample-wise Uncertainty-Guided Regularization (SUGR), which adaptively assigns sample-wise uncertain weights to samples to optimize training. Extensive experiments demonstrate that MCUR consistently outperforms existing methods, achieving average F1 gains of 2.2% on MOSI, 2.67% on MOSEI, and 4.37% on IEMOCAP.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Incomplete Learning, Multimodal Emotion Recognition, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yyATOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MCUR, a teacher–student framework for multimodal emotion recognition with missing modalities. It introduces modality-combination- and category-based contrastive learning (MCB-CL) to regulate representations, and sample-wise uncertainty-guided regularization (SUGR) to weight task and distillation losses using teacher–student uncertainty differences. Experiments on MOSI, MOSEI, and IEMOCAP report improvements across fixed and random missing-modality settings.

Critical strengths

The paper tackles an important practical problem: emotion recognition when different samples have different available modalities. It considers both how representations change with missing inputs and how a full-modality teacher should guide a student with incomplete inputs. The experiments cover three common benchmarks and a range of missing-modality settings.

Critical weaknesses
  1. The main contributions are not clearly distinguished from existing work. The framework builds on contrastive learning, representation alignment, and loss weighting, all of which have been studied before. The paper needs to explain more clearly what is new in the proposed objectives and why these particular components are needed.

  2. The baseline comparisons are limited. Most methods in the main comparison are from 2023–2024, although the related work discusses newer approaches. Without comparisons to relevant recent methods under the same evaluation settings, it is difficult to judge how competitive MCUR is. The MOSI results are also placed in the appendix rather than alongside MOSEI and IEMOCAP in the main results table.

  3. It is unclear whether MCB-CL actually uses modality-combination information in its default setting. In Eq. (5), the default values are μ1=μ2=1. Substituting Eqs. (2)–(4) appears to cancel the modality-combination terms, leaving a contrastive objective based only on emotion labels, provided the sample sets are defined consistently. If the implementation follows these equations, what part of the loss still enforces modality-combination-specific alignment? The authors should clarify this point and explain the difference from ordinary label-based contrastive learning.

  4. The entropy difference used in SUGR does not necessarily indicate reliable teacher supervision. A teacher and student can predict different classes with equally high confidence and still have the same entropy. In that case, Eq. (9) gives a zero uncertainty weight despite the disagreement. The absolute difference also treats cases where the teacher is more uncertain and less uncertain than the student in the same way. It would help to explain why this is a suitable weighting signal and to compare it with a measure that directly captures prediction disagreement.

  5. The manuscript appears not to strictly follow the ICLR formatting requirements. The formatting should be checked against the official template.

Questions for authors

Please refer to the concerns in the Critical Weaknesses.

Reviewer_gavxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MCUR, a teacher–student framework for Multimodal Emotion Recognition (MER) with missing modalities. The teacher uses complete modalities, while the student learns from incomplete inputs. MCB-CL constructs positive pairs based on emotion category and modality combination. SUGR weights the task and distillation losses using teacher–student differences in predictive entropy or squared error. Experiments under fixed and random missing settings on MOSI, MOSEI, and IEMOCAP report average F1 gains of 2.20, 2.67, and 4.37 percentage points, respectively, alongside component ablations and robustness analyses.

Critical strengths
  1. The motivation makes sense: text-only and audio-visual inputs may offer different amounts of evidence for the same emotion. The authors account for the available modality combination when choosing positive pairs and use a full-input teacher to guide the student. At inference, the model works directly with the available modalities without first reconstructing what is missing.

  2. The experiments cover three datasets and 14 settings, with component ablations in Table 2 and contrastive learning comparisons in Table 3. Table 12 also adds MCB-CL and SUGR separately to CorrKD and MMANet, so the gains are not just from changing the whole backbone. The appendix includes multiple seeds, noise, imbalance, teacher bias, and MOSEI-to-MOSI transfer. Average F1 rises from 74.13 to 76.80 on MOSEI and from 73.28 to 77.65 on IEMOCAP compared with the strongest average-F1 baseline.

Critical weaknesses
  1. I am not fully convinced that the quantity used by SUGR captures uncertainty. For regression, it is prediction error. For classification, similar teacher and student predictive entropy can also mean that both models are confidently wrong. SUGR could then give such a sample a small weight. Better Brier score and NLL are encouraging, but they do not tell us whether this weighting has the intended effect. A comparison with simple error/confidence weighting to understand what the proposed design adds is needed.

  2. The experiments cover a lot of settings, but there is still a gap between simulated and real missing modalities. Most tests apply simulated modality dropout to originally complete data, and teacher training also uses complete modalities. In practice, missingness may depend on the speaker, device, or scene rather than occur randomly. The noise, imbalance, and cross-dataset transfer experiments help, but they do not directly show whether the gains hold under this kind of non-random missingness.

Questions for authors
  1. What weights does SUGR assign when both the teacher and student are confidently wrong? Could you compare it with simple error/confidence weighting using the same backbone and teacher? How much of the gain comes specifically from uncertainty-guided supervision.

  2. Do MCUR's gains hold when missingness depends on the speaker, device, or emotion label, rather than random dropout? Do any existing results address this? If not, please make the gap between simulated missingness and real applications clearer.

Reviewer_LBXzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MCUR for multimodal emotion recognition under missing-modality conditions. It combines Modality Combination-Based and Category-Based Contrastive Learning (MCB-CL) with Sample-wise Uncertainty-Guided Regularization (SUGR). The results on MOSI, MOSEI, and IEMOCAP are promising, but several concerns remain regarding the mathematical formulation and experimental validation.

Critical strengths
  1. The paper addresses a practical problem in multimodal emotion recognition with missing modalities.

  2. The experiments cover three datasets and various missing-modality settings, with generally competitive results and reasonably extensive ablations.

Critical weaknesses
  • According to Eqs. (2)–(5), when the default hyperparameters μ₁ = μ₂ = 1 are used, the modality-combination terms cancel out algebraically, reducing MCB-CL to a class-based supervised contrastive objective. This appears inconsistent with the claimed modality-aware mechanism. The authors should clarify this issue and explain the performance gap between MCB-CL and SupCon in Table 3.

  • SUGR uses the absolute entropy difference between teacher and student predictions as a loss weight. However, two predictions can have identical entropy even when they strongly disagree, resulting in a zero weight. Furthermore, using prediction error as uncertainty in regression tasks requires further justification. Comparisons with established uncertainty-weighting strategies would strengthen the method.

  • The overall loss in Eq. (15) differs from Eq. (23) in the appendix, where an additional VIB loss is included. Moreover, the paper requires at least one available modality per sample, implying a maximum missing rate of 2/3 for three modalities, yet experiments report a missing rate of 0.7. These inconsistencies should be clarified.

  • The manuscript mainly uses Latin Modern fonts rather than the Times New Roman typeface preferred by the official ICLR template. The title and section formatting should also be checked.

Questions for authors

see Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.