acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Affective Representation Learning: A Technical Evolution from Feature Disentanglement to Semantic Alignment

Abstract

Multimodal affective representation learning aims to learn discriminative, transferable, and robust affective representations from heterogeneous information, including text, speech, visual signals, physiological signals, and eye movements. This review systematically surveys the field from four perspectives: emotion theories, modality-specific information, representation-learning methods, and application-oriented evaluation. Existing approaches are organized into five families: cross-modal alignment and representation disentanglement, cross-modal interaction and dynamic fusion, context and dialogue-structure modeling, self-supervised and robust learning, and large-model-based and prompt-based learning. We further analyze these methods through six affective representation metrics: cross-modal comparability, complementarity, redundancy control, disentanglement of irrelevant factors, robustness to missing modalities, and sensitivity to dynamics, and examine their roles in affect detection and recognition, affective reasoning, affective response and support, and cross-task generalization. Finally, we summarize current evaluation practices and their limitations, with the aim of moving multimodal affective computing beyond single-task performance toward a more systematic assessment of affective representation quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Affective Representation Learning, Multimodal Affective Computing, Multimodal Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_mM1eOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper reviews multimodal affective representation learning from the perspectives of emotion theories, modality-specific information, representation learning methods, and downstream applications. It categorizes existing methods into five families and introduces six properties for assessing affective representations. The survey discusses representative corpora, evaluation practices, and challenges related to robustness and generalization.

Critical strengths
  1. The review considers different emotion theories, input modalities, learning paradigms, and downstream applications.
Critical weaknesses
  1. Previous reviews (see Journals Q1'25-26) already cover emotion theories, multimodal representations, fusion, alignment, disentanglement, evaluation, and generalization. The Introduction does not provide a substantive comparison establishing what new scientific knowledge this survey contributes beyond these studies.
  2. The five families in Section 5 and Tables 1 and 3 describe established methodological categories. Their boundaries are not clearly defined. For example, contrastive alignment, dynamic fusion, and robust learning can coexist within the same architecture. The paper provides no systematic classification criteria or analysis of overlapping methods. Consequently, the taxonomy offers limited methodological insight beyond existing classifications.
  3. The title suggests a technical progression from feature disentanglement to semantic alignment. However, the paper does not establish this progression through a chronological analysis, methodological dependencies, or comparative evidence. Sections 3-5 describe existing methods as parallel categories rather than demonstrating a meaningful technical evolution.
  4. The Introduction identifies six affective representation metrics, including cross-modal comparability, complementarity, redundancy control, disentanglement, robustness, and sensitivity to dynamics. However, these properties lack operational definitions, measurement procedures, and validation criteria. The paper therefore does not establish a reproducible framework for evaluating representation quality.
  5. Tables 4-7 evaluate only four dimensions despite the six metrics introduced in the paper. Redundancy control and sensitivity to dynamics are omitted. Moreover, the qualitative assignments are largely identical across different application settings. The criteria for distinguishing dominant and auxiliary roles are not provided. These tables do not substantiate the claimed application-specific analysis.
  6. Section 2 and Appendix A report 260 retrieved records and 146 final references. However, the complete database-specific search queries, search dates, exclusion records, and screening decisions are unavailable. The authors also report supplementary retrieval using ChatGPT, Claude, and Gemini without specifying the retrieval or verification procedures. The reported selection process cannot be independently reproduced.
  7. Section 6.5 is presented as an analysis of experimental findings but contains no systematic quantitative comparison of methods. It provides neither comparable benchmark results nor evidence linking specific representation properties to observed performance. Consequently, the paper does not demonstrate that its framework produces additional empirical insights beyond conventional task-level evaluations.
  8. The review includes several text-only datasets for affective reasoning and emotional support without clearly separating unimodal findings from evidence about multimodal representations. Conversely, eye tracking is presented as a relevant modality in Section 4 but is not represented in the dataset comparison in Table 8.
  9. Table 8 reports heterogeneous dataset statistics and annotation schemes without standardized information on experimental protocols, data partitions, or evaluation comparability. Section 6 emphasizes cross-task generalization but does not systematically distinguish cross-dataset, cross-domain, cross-lingual, and missing-modality evaluations. As a result, the survey provides limited guidance for determining which published results are directly comparable.
Questions for authors
  1. What specific scientific conclusions distinguish the proposed framework from the closest previously published surveys?
  2. What objective criteria determine the assignment of existing methods to the five proposed methodological families?
  3. What evidence establishes the claimed technical progression from feature disentanglement to semantic alignment?
  4. How can the six proposed representation properties be measured quantitatively?
  5. What criteria were used to assign dominant and auxiliary roles in Tables 4-7?
  6. Can the authors provide the complete database-specific search protocol and screening records supporting the selection of 146 references (see PRISMA)?
  7. What comparative evidence demonstrates that the proposed representation-centered framework provides additional insights beyond existing task-level evaluations?
Reviewer_M9jCOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper surveys multimodal affective representation learning by connecting psychological emotion theories (categorical, dimensional, appraisal-based, and open-ended descriptions) with heterogeneous inputs including text, speech, visual signals, physiological signals, and eye movements. Methodologically, it organizes existing literature into 5 paradigm families: cross-modal alignment and disentanglement, dynamic fusion, context and dialogue-structure modelling, self-supervised/robust learning, and large-model-based learning. Then, it maps these techniques onto 4 application scenarios: affect detection and recognition, affective reasoning, affective response and support, and cross-task generalization. The authors analyzed 146 references and summarized standard metrics and benchmark performance across established datasets such as CMU-MOSEI, IEMOCAP, RECCON, and EmpatheticDialogues.

Critical strengths

The manuscript is well-structured, clearly written, and logically organized. Figs. 1-7 and Tabs. 1-8 provide clear visual and tabular taxonomies that effectively bridge theoretical foundations, methodological families, and downstream application scenarios. The manuscript connects fundamental psychological emotion theories and 5 methodological paradigms, providing a structured framework for evaluating affective computing systems beyond isolated tasks. Rather than judging models solely on downstream classification accuracy, the paper highlights critical representation properties including factor disentanglement, cross-modal comparability, dynamic sensitivity, and robustness under missing modalities.

Critical weaknesses

The paper compiles a broad selection of recent advances and emphasizes representation-level quality metrics such as disentanglement of irrelevant factors and robustness to missing modalities over raw task accuracy. However, because it is a literature survey without novel empirical experiments or meta-analytic benchmarks, it reorganizes existing knowledge rather than discovering new foundational principles. As a review paper submitted to a conference, it introduces no new algorithmic architectures, mathematical proofs, or primary empirical benchmarks. Furthermore, the experimental section merely restates published evaluation results across non-standardized experimental setups. The search protocol relies on unscripted retrieval via commercial LLMs with keyword searches, which lacks deterministic reproducibility and risks introducing selection bias into the included corpus. Almost 10% of references are pre-prints.

Questions for authors
  1. Given that commercial LLMs are non-deterministic and update continuously, how do you ensure the exact reproducibility of your literature selection process, and what specific prompt templates or exclusion heuristics were used during LLM-assisted search?
  2. In Section 6, performance trends are drawn across studies with non-identical dataset splits and evaluation protocols. Could the authors conduct a unified quantitative meta-analysis or re-evaluate representative open-source models from each of the five paradigm families under a single standardized benchmark protocol, such as CMU-MOSEI or IEMOCAP?
1 comment

The authors did not submit a rebuttal to address my comments or respond my questions. Therefore, I am changing my overall rating to 1.

Reviewer_e4f5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This is a survey of multimodal affective representation learning. It goes over emotion models and the cues carried by text, speech, vision, physiological signals and eye movements, and groups existing methods into five families (alignment/disentanglement, dynamic fusion, context and dialogue modeling, self-supervised and robust learning, and LLM/prompt-based methods). These are then linked to four application settings: recognition, affective reasoning, response and support, and cross-task generalization. The authors argue that methods should be judged by the quality of their representations rather than task scores alone, and propose six "affective representation metrics" for this.

Critical strengths

The idea of looking at representation properties, such as disentangling speaker identity or robustness to missing modalities, instead of only accuracy is reasonable. The modality coverage is broad and includes physiological signals and eye tracking, and Tables 2 and 8 can be a handy starting point for newcomers.

Critical weaknesses
  1. The main contribution is not really developed. The abstract says the methods are analyzed "through six affective representation metrics". Apart from the list in the contributions (lines 139-142), however, each metric only gets a one-line description in Figure 2, and two of them even have different names there ("Modal Redundancy Suppression", "Dynamic & Context-Sensitive"). The metrics are not formalized, not measurable, and not used to analyze any of the surveyed methods. Tables 4-7 then use a different set of four dimensions. I also could not find the "evolution from feature disentanglement to semantic alignment" that the title promises, and the phrase appears only once more, at line 97.
  2. The main text has very little survey content. Section 5 is supposed to be the core of the paper, but it is two paragraphs plus Table 1, and Sections 3 and 4 are about as short. Almost everything is pushed to the appendix, so the main paper does not actually compare the methods in any technical depth.
  3. There is no experimental analysis. The introduction mentions "an integrated experimental analysis" (lines 161-162), but Section 6.5 has no numbers and mostly repeats 6.1-6.4. I would expect at least a summary of reported results on CMU-MOSI/MOSEI, MELD or IEMOCAP, ideally with complete vs. missing modalities, to back up the claims about robustness.
  4. Several factual problems. The paper says 146 references were included (line 178, Appendix A, Fig. 3), but the bibliography has 105 entries. Figures 2 and 7 use numeric citation labels such as [134] and [143], which do not exist in this paper's author-year bibliography. Fig. 5(b) is attributed to Yang et al. (2022), but the tri-subspace design with a submodally-shared encoder and a decoupling supervisor is the TSD framework of Meng et al. (2026), which is also in the bibliography. Some citations do not fit the claims. Lamme (2006), a paper on consciousness, is cited for cultural and contextual effects on visual signals (line 229), and ResNet/ViT are cited for facial expression and posture cues (lines 219-220).
  5. The application tables say little. Tables 6 and 7 are identical, and Tables 4 and 5 each differ from them in only one cell, even though Appendix E.4 says alignment and robustness are "particularly prominent" for cross-task generalization.
  6. Uneven coverage. Some representative works in the paper's own categories are missing, such as MMIN (Zhao et al., 2021) for missing modalities, Self-MM (Yu et al., 2021) for self-supervised learning, and Emotion-LLaMA (Cheng et al., 2024) for MLLM-based emotion understanding.
  7. Presentation issues. Fig. 5 reuses published figures without saying so: panel (a) is Figure 2 of the MISA paper with the feature-extraction column cropped, and panel (b) comes from Meng et al. (2026). There are typos in figures, namely "Respresentation" (Fig. 1), "pormpt" (Fig. 3) and "Discerete" (Fig. 4). The text in Fig. 2 is too small to read. In Table 8, the first row label overlaps with the CPED row, and the note defines E = eye tracking even though no eye-tracking dataset is listed.
Questions for authors
  1. Can the six metrics be made measurable, e.g., as a disentanglement score, the performance drop under modality dropout, or cross-modal retrieval alignment? Could they be applied to a few representative methods (e.g., MISA, MulT, UniMSE and an MLLM baseline) on a common benchmark? A comparison like this would most change my assessment.
  2. How were the filled/empty circles in Tables 4-7 decided, and why are Tables 6 and 7 identical?
  3. Please reconcile the 146 included references with the 105 bibliography entries. Please also fix the numeric citations in Figures 2 and 7 and the attribution of Fig. 5(b).
  4. What concrete trends support the "from feature disentanglement to semantic alignment" narrative? A timeline or a quantitative trend analysis would help. Also, how were the papers suggested by LLM-assisted retrieval (Appendix A) verified?
  5. The paper cites Wang et al. (2026), "A survey of representation learning for multi-modal emotion recognition", throughout, and that survey also organizes the field around six representation-learning problems. Could you add it to the related surveys at lines 90-96 and briefly explain what this paper adds beyond it, e.g., the wider modality coverage and the reasoning and support settings? (I understand this is contemporaneous work, and it does not affect my rating.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.