acceptodds
Withdrawn from ICLR 2027

EmoGPT: Towards Explainable Multimodal Emotion Recognition with Cross-Modal Chain-of-Thought Supervision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding human emotions from multimodal signals requires reasoning over potentially conflicting cues across facial expressions, vocal prosody, and language. However, existing Multimodal Large Language Models (MLLMs) rely primarily on categorical supervision or unstructured descriptions, limiting their ability to perform interpretable cross-modal reasoning, especially in cases of semantic–prosodic inconsistency such as sarcasm. In this work, we introduce structured cross-modal Chain-of-Thought (CoT) supervision for multimodal emotion recognition. We present EmoGPT, a framework that learns to generate emotion predictions grounded in explicit reasoning over visual, acoustic, and textual evidence. To enable this, we construct ECoT-65K, a large-scale Emotional Chain-of-Thought dataset, designed to capture modality-specific evidence and their interactions. Beyond supervision, we propose two inductive mechanisms to improve multimodal reasoning. First, Affective Visual Routing (AVR) introduces a mixture-of-experts projector that adaptively routes visual tokens based on emotion-relevant semantics. Second, Emotional Saliency Bias (ESBias) introduces zero-initialized, layer-specific Q/K/V/O offsets that complement LoRA with emotion-sensitive affine adaptation. Extensive experiments on nine benchmarks under three modality configurations show that EmoGPT achieves the highest mean accuracy in every setting, improving the AVT mean by 4.51 points over the previous state of the art; notably, fine-tuning on ECoT-65K alone already surpasses the previous state of the art without any architectural change. A human audit further shows that 94.3% of the retained reasoning annotations are rated as grounded in multimodal evidence, advancing multimodal emotion recognition toward interpretable and robust cross-modal reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Emotion Recognition, Multimodal Large Language Models, Cross-modal Reasoning, Chain-of-Thought Supervision, Affective Computing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sNcdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors construct a new dataset, ECoT-65K, from multiple existing datasets to train their proposed EmoGPT. They further propose a MoE-based emotion-processing projector and a learnable bias, ESBias. Through ablation and parameter studies, they verify that the emotion-processing projector and the bias improve model performance.

Critical strengths
  1. Applying MoE to handle information from different modalities is logically sound. Routing-based processing of multimodal content to effectively filter modality information is already a consensus in the field.
  2. The introduction of CoT effectively improves the interpretability of sentiment analysis results and injects external knowledge from reasoning chains into the model.
Critical weaknesses
  1. ESBias adds a learnable bias to the attention matrix computation. In my view, this bias could be introduced simply by enabling the linear layer in the W matrix. Therefore, the novelty of this module is debatable. Moreover, the main function of ESBias is to add more learnable parameters to attention computation; the claimed learning of preferences is not supported by any experiment. It is thus unclear whether the gains come from the additional parameters, from some unintended perturbation, or from genuinely learned preference information.
  2. ECoT-65K is built by further annotating existing datasets, with annotations generated by large models. LLM-generated labels contain hallucinations and errors, while only part of them were manually verified. In sentiment analysis, sentiment labels require careful expert human verification to avoid unfair judgments and incorrect labels. ECoT-65K does not appear to meet this standard.
  3. The comparison experiments are unfair. Both Qwen-Omni (finetuned) and EmoGPT were fine-tuned on ECoT-65K, while it is unknown whether the other baselines had access to this dataset. Given the superior performance of Qwen-Omni (finetuned), the reported gains may be attributable to the generated CoT content in ECoT-65K.
Questions for authors
  1. The authors make many claims in the motivation and contribution sections without corresponding experimental verification, e.g., Lines 30–32, 51–53, and 199–203. Subsequent experiments also do not further demonstrate whether the problems raised in the motivation are addressed by existing methods.
  2. The experiments report no means, variances, or statistical tests such as t-tests. The current results cannot rule out selective reporting.
  3. The authors should further evaluate the label correctness of ECoT-65K to support further research by the community.
Reviewer_eQyrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a key limitation in multimodal emotion recognition (MER): existing MLLMs rely primarily on emotion labels or unstructured textual descriptions, lacking structured supervision of the cross-modal reasoning process. This limitation is particularly important when emotional cues across modalities conflict, such as when semantic content contradicts vocal prosody in sarcasm. To address this, the authors introduce structured cross-modal chain-of-thought (CoT) supervision, enabling models to predict emotion labels while explicitly reasoning over visual, acoustic, and textual evidence. Specifically, they construct ECoT-65K, a dataset of 65,370 video-reasoning pairs drawn from seven public corpora, and propose two architectural enhancements: Affective Visual Routing (AVR), a four-expert top-2 sparse MoE visual projector, and Emotional Saliency Bias (ESBias), which introduces zero-initialized Q/K/V/O biases into the thinker attention layers. Fine-tuning Qwen2.5-Omni-7B on ECoT-65K alone improves AVT mean accuracy from 67.16% to 75.41%. Incorporating AVR and ESBias further improves performance, yielding an additional 3.03-point average gain across AT, VT, and AVT settings and raising AVT mean accuracy to 79.28%.

Critical strengths
  1. The paper raises a useful idea. Current emotion recognition models rely mainly on categorical supervision. Their training data lacks process-level reasoning traces.
  2. The models cannot clearly show how an emotion judgment is formed.The authors extract samples from existing datasets. They re-annotate these samples with structured cross-modal CoT. The resulting ECoT-65K provides a useful data resource for follow-up research. Experiments show that fine-tuning Qwen2.5-Omni-7B on this dataset alone raises AVT mean accuracy from 67.16 to 75.41
  3. The paper is well organized and easy to follow.
Critical weaknesses
  1. The motivation behind the proposed architectural designs is insufficiently justified. The paper does not clearly identify the specific limitations that motivate AVR and ESBias. In particular:
  • AVR: Why is a sparse MoE necessary for visual emotion features? Why is routing applied only to visual tokens, rather than audio or text, which also contain diverse emotional cues?
  • ESBias: Although additive adaptation is intuitively reasonable, the demonstrations of what limitations of standard LoRA it addresses or why such additional parameters are necessary are not satisfactory.
  1. The novelty of the structured CoT supervision is limited and insufficiently differentiated from prior work.
  • Closely related studies, such as EmoThinker (CVPR 2026) [1] and EMO-CoT (IEEE Transactions on Affective Computing 2026) [2], have already explored structured multimodal emotion reasoning and fine-grained evidence-based annotations. Their differences from ECoT-65K are not adequately discussed.
  • Beyond emotion recognition, LLaVA-CoT (ICCV 2025) [3] and VideoEspresso (CVPR 2025) [4] have established similar paradigms for generating structured CoT annotations from existing multimodal data.
  • The paper does not sufficiently explain what is fundamentally new about its annotation strategy or why existing CoT annotation methods cannot be directly adapted to this task.
  1. Important methodological details and design justifications are missing.
  • For AVR, the intended specialization of each expert, initialization strategy, and training dynamics are not sufficiently explained. It is unclear whether the experts actually learn distinct emotion-relevant representations.
  • For ESBias, the roles and necessity of the individual Q/K/V/O offsets remain unclear. More detailed explanations and component-wise analyses are needed to justify the design choices.
  1. The experimental evaluation is insufficient to support the main claims.
  • Insufficient baselines: The paper lacks label-only and caption-style supervision baselines with matched data sources, sample counts, and training budgets. It also lacks direct comparisons with existing CoT annotation strategies. Therefore, the benefits of the proposed structured supervision remain unclear.
  • Insufficient module validation: AVR lacks a parameter-matched dense projector baseline, expert specialization analysis, and routing statistics. ESBias lacks comparisons against stronger or carefully tuned LoRA configurations and component-wise ablations.
  • Limited generalizability: Experiments are conducted only on Qwen2.5-Omni-7B. Evaluation on additional recent open-source backbones is needed to determine whether the improvements generalize.
  • Limited reliability analysis: Multiple runs, confidence intervals, and statistical significance tests are missing. The human audit evaluates annotation quality rather than the faithfulness of model-generated reasoning. Potential overlap between training and evaluation datasets is also not explicitly examined.
  1. The writing and presentation need minor improvements. Model naming and abbreviation usage should be more consistent. Figure 1 is difficult to interpret because of overlapping radar plots, and Table 2 would benefit from improved alignment and formatting. Several long sentences and minor notation inconsistencies should also be revised.

Ref:

[1] EmoThinker: Advancing Visual-Acoustic Emotion Analysis via Structural Token Selection and Chain-of-Thought Reasoning. CVPR 2026.

[2] Chain-of-Thought Driven Multimodal Emotion Reasoning and Evaluation. IEEE Transactions on Affective Computing, 2026.

[3] LLaVA-CoT: Let Vision Language Models Reason Step-by-Step. ICCV 2025.

[4] VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning. CVPR 2025.

Questions for authors

See Weaknesses for details.

Reviewer_cFuYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies emotion recognition using textual, acoustic, and visual cues, together with the generation of corresponding explanations. The authors construct ECoT-65K, containing 65,370 structured reasoning annotations derived from seven public corpora, and augment Qwen2.5-Omni-7B with an Affective Visual Routing module (AVR) and an Emotional Saliency Bias module (ESBias). Experiments cover nine benchmarks and three input-modality combinations, reporting improvements in average benchmark scores, module ablations, and a human audit of 70 training annotations.

Critical strengths
  1. The evaluation covers multiple benchmarks, languages, and input-modality combinations, with the full model achieving strong average results. Individual and joint module ablations also provide evidence of effectiveness under the evaluated training setup.
  2. ECoT-65K organizes multimodal evidence, reasoning, and final judgments into a unified supervision format, combining automatic filtering with a human audit. It has potential value as a resource for subsequent research on emotion reasoning.
Critical weaknesses
  1. The reliability of generated explanations lacks direct validation. The human audit covers only 70 training annotations, with an average of 64.5% of ratings marked as fully supported across the six evidence-and-reasoning dimensions. Inter-annotator agreement is not reported, and the specific shortcomings and their severity in partially supported annotations are not analyzed. Furthermore, the paper does not systematically evaluate the factual correctness of explanations generated on test examples or their faithfulness to the model’s decisions. The corresponding claims therefore remain insufficiently supported.

  2. Direct evidence linking the modules’ design motivations to their actual behavior is missing. Although AVR and ESBias receive emotion-task supervision and improve classification performance, no direct analysis validates the proposed expert specialization or emotional saliency effects. The connection between their intended mechanisms and learned behavior therefore remains unestablished.

Questions for authors
  1. Is there a direct comparison of label-only supervision, ordinary descriptions, and structured reasoning supervision using the same backbone, the same source examples, and comparable training settings? After accounting for training and generation costs, which gains can be attributed specifically to the reasoning structure?
  2. Could the authors report the distribution of the 70 human-audited examples, inter-annotator agreement, and the criteria used to assess AU and prosodic descriptions?
  3. Have the factual correctness and faithfulness of generated explanations, as well as their advantages over baselines, been evaluated on unseen examples and a subset with cross-modal conflicts?
  4. Are there routing or feature-response analyses, comparisons with and without the learned biases, or intervention results demonstrating that AVR and ESBias actually use the claimed emotion-relevant cues?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.