acceptodds
Under review as a conference paper at ICLR 2027

From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding

Abstract

Recent multimodal large language models (MLLMs) increasingly incorporate explainable reasoning for emotion understanding. However, reasoning based mainly on observable affective cues, without deeper theory-guided analysis of how these cues relate to emotion formation, may reduce emotion understanding to statistical associations between cues and emotion categories. This reliance on superficial associations can give rise to the Clever Hans effectClever Hans Effect: A system exhibits seemingly intelligent behavior but actually relies on hidden cues, data biases, or superficial associations rather than genuinely understanding the mechanisms underlying the task., where model predictions depend on surface-level correlations rather than a genuine understanding of why emotions arise. Consequently, such shortcuts often become unreliable when affective cues are implicit, conflicting across modalities, linguistically misleading, or contaminated by redundant details. %visually absent In contrast, human emotions are shaped by how individuals interpret and evaluate surrounding events beyond observable cues. Inspired by appraisal theories of emotion, we formulate multimodal emotion understanding as a progression from perception to cognitive appraisal, and introduce a dataset, a model, and a benchmark to support this paradigm. 182 CogEmo-40K is a large-scale instruction-tuning dataset constructed through a perception-to-appraisal pipeline, where models generate evidence-grounded reasoning across six appraisal dimensions including goal congruence, coping potential, accountability, future expectancy, arousal, and action tendency. 183 CogEmo-MoE is a sparse MLLM that explores cognitive appraisal-guided emotion reasoning with Mixture-of-Experts architectures. 184 CogEmo-Bench evaluates free-form model outputs by jointly measuring appraisal evidence quality and emotion recognition performance through the Appraisal Evidence Quality Score (AEQS) and emotion classification metrics. Extensive experiments show that our paradigm not only leads CogEmo-Bench (CogEmo-MoE: \(56.30\!\rightarrow\!70.60\) AEQS vs. GPT-5.2; \(63.10%\!\rightarrow\!71.43%\) accuracy and \(52.19%\!\rightarrow\!57.17%\) M-F1 vs. Qwen3-Omni), but also exhibits strong cross-domain generalization, highlighting the effectiveness of perception-to-appraisal reasoning in moving beyond surface-level cue-label associations toward more reliable multimodal emotion understanding.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal emotion understanding, cognitive appraisal, multimodal large language models, affective computing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_M11WOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a perception-to-appraisal paradigm for MERg, in which observable evidence is interpreted through six cognitive appraisal dimensions before predicting an emotion. It contributes CogEmo-40K for instruction tuning, CogEmo-Bench with AEQS for appraisal-oriented evaluation, and CogEmo-MoE, a compact model using interleaved sparse experts and progressive training. Experiments on the proposed benchmark and multiple external emotion datasets demonstrate improvements in both appraisal evidence quality and emotion recognition.

Critical strengths
  1. The motivation is reasonable, and the overall contribution is relatively solid, covering a training dataset, an appraisal-oriented evaluation benchmark, and a corresponding model.

  2. The paper is generally well presented, with a clear progression from motivation to data construction, model design, and evaluation. The experiments are also fairly extensive, including comparisons with diverse MLLMs, architectural ablations, modality analysis, and cross-domain evaluation.

Critical weaknesses
  1. My main concern is how appraisal is operationally framed for both training and evaluation. Although the main text names the six appraisal dimensions and outlines the construction pipeline, their definitions and demonstrations are absent (from the main text). These choices are central to the paper because CogEmo-40K and CogEmo-Bench use the same appraisal framework and closely aligned construction protocols. It is therefore difficult to determine whether the model learns deeper cognitive appraisal or primarily learns to reproduce the expected dimension-specific language and emotion-correlated appraisal patterns.

  2. (Minor) The technical contribution of CogEmo-MoE appears relatively limited. The method largely builds on MoE-LLaVA by copying FFN weights into sparse experts and applying progressive supervised fine-tuning, making it closer to an architectural and training adaptation.

Questions for authors
  1. Could the authors summarize more concretely what constitutes each appraisal dimension in the training targets and evaluation references, and clarify how closely the two construction procedures are aligned? More importantly, what evidence demonstrates that the model performs evidence-grounded appraisal reasoning rather than imitating recurring appraisal phrases or emotion-specific templates learned from CogEmo-40K?
  2. Could the authors perform some RL-based optimization on baselines and analyze how they contribute to this new reasoning paradigm?
4 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer M11W

Weakness 1 & Question 1: How are the six appraisal dimensions operationalized, and does the model learn appraisal reasoning rather than recurring templates?

Response. Thank you for this important comment. Detailed definitions of the six appraisal dimensions are provided in Appendix Table 7, with cross-model examples in Appendix E.4. Their operational meanings are summarized below. The generation protocol and human verification procedure are detailed in Appendices B.2 and B.3, respectively. We will add this summary and clearer appendix pointers to the main text.

For your convenience, we have copied Table 7 below.

DimensionOperational meaning
Goal CongruenceWhether the situation supports, obstructs, or is irrelevant to the subject’s goals, needs, or expectations.
Coping PotentialThe subject’s perceived ability to manage, control, or adapt to the situation.
AccountabilityWhether responsibility is attributed to oneself, another person, or no identifiable agent.
Future ExpectancyThe anticipated development and consequences of the situation, including no salient future expectation.
ArousalThe level of activation supported by vocal, facial, and bodily cues.
Action TendencyThe behavioral orientation implied by the appraisal, such as engaging, avoiding, or confronting.

CogEmo-40K and CogEmo-Bench share the same cognitive appraisal framework and description-level perception principles to ensure consistency between training and evaluation objectives, but they differ in the strength of quality control. CogEmo-40K adopts a human-in-the-loop verification and correction process, whereas CogEmo-Bench uses held-out samples not involved in Stage-3 training, and its evaluation references are further reviewed and manually refined at the sample level by researchers with prior experience in affective computing. Therefore, CogEmo-Bench does not simply reuse the training annotations.

We also conducted a grounding stress test to examine whether AEQS primarily rewards recurring appraisal phrases or emotion-specific templates. For each test sample, we constructed four candidate explanations: Matched, the appraisal corresponding to the current sample; Paraphrased Matched, a meaning-preserving paraphrase of the matched appraisal with different surface wording; Same-emotion Mismatch, an appraisal transferred from another sample with the same emotion label but different event and appraisal content; and Random Mismatch, an appraisal transferred from an unrelated sample.

Candidate AppraisalAEQS ↑Δ vs. Matched
Matched90.31–
Paraphrased Matched83.77−6.54
Same-emotion Mismatch65.39−24.92
Random Mismatch60.23−30.08

The results show that AEQS remains relatively high for Paraphrased Matched explanations despite substantial changes in surface wording, while it drops sharply for Same-emotion Mismatch and Random Mismatch. In particular, the 24.92-point drop under Same-emotion Mismatch, where the emotion label is unchanged, suggests that the evaluation is sensitive to instance-specific appraisal content and context rather than mainly rewarding emotion-specific templates or fixed phrasing.

Taken together, these results show that AEQS is sensitive to instance-specific appraisal content beyond recurring surface phrases. Evidence concerning the learned model is complementary: under Appraisal + Emotion supervision, CogEmo-MoE improves Accuracy from 68.35% to 71.43% and Macro-F1 from 53.04% to 57.17% relative to Emotion-Only training; the cross-domain evaluation in Appendix E.2 further supports the utility of cognition-grounded supervision beyond CogEmo-Bench. These results support the proposed training paradigm. We will make the operational definitions and these distinctions more explicit in the revision.

AuthorsOpenReview ↗

[2/3] Response to Reviewer M11W

Weakness 2: Is the technical contribution of CogEmo-MoE limited to an architectural and training adaptation of MoE-LLaVA?

Response. Thank you for this observation. We agree that the MoE architecture itself is not intended to be a fundamentally new MoE design. CogEmo-MoE builds on the MoE-LLaVA framework, while our main methodological focus is on adapting sparse conditional computation to the proposed Perception → Appraisal → Emotion paradigm through progressive multimodal alignment, perception enhancement, and cognitive appraisal tuning.

Our additional controlled ablation further clarifies this distinction. Specifically, we conducted a 2 × 2 ablation while keeping Stages 1 and 2 fixed and initializing all models from the same Stage-2 checkpoint. We varied only two factors in Stage 3: (1) the supervision target, comparing Emotion Only with Appraisal + Emotion, and (2) the architecture, comparing Dense with 4-expert Top-2 MoE. The source samples, training budget, and hyperparameters were otherwise kept the same.

Stage-3 SupervisionArchitectureAEQS ↑Emotion Acc. ↑Macro-F1 ↑
Emotion OnlyDense2.1066.7453.60
Emotion Only4E Top-2 MoE1.9868.3553.04
Appraisal + EmotionDense70.4867.2653.27
Appraisal + Emotion4E Top-2 MoE70.6071.4357.17

The large AEQS increase from Emotion Only to Appraisal + Emotion shows that cognitive appraisal capability is primarily established by appraisal-guided supervision. In contrast, under the same Appraisal + Emotion supervision, replacing Dense with 4E Top-2 MoE changes AEQS only from 70.48 to 70.60 (+0.12), while improving emotion Accuracy from 67.26% to 71.43% (+4.17) and Macro-F1 from 53.27% to 57.17% (+3.90). This indicates that the sparse MoE provides a complementary architectural benefit, particularly for final emotion discrimination, rather than being the primary source of cognitive appraisal reasoning.

We therefore agree that our original wording may overemphasize MoE specialization. In the revision, we will narrow this claim and more clearly position CogEmo-MoE as an architectural realization of the proposed cognitive appraisal reasoning framework, rather than presenting the MoE mechanism itself as the primary technical novelty.

AuthorsOpenReview ↗

[3/3] Response to Reviewer M11W

Question 2 : Could RL-based optimization of the baselines further contribute to the proposed reasoning paradigm?

Response. Thank you for this valuable suggestion. We agree that reinforcement learning (RL) is a promising direction for further improving cognitive appraisal reasoning. However, we did not introduce RL optimization into the current comparison for two main reasons.

First, adding RL to the baselines would simultaneously introduce additional factors such as the reward design, optimization objective, and training budget, making it more difficult to isolate the contribution of cognitive appraisal supervision itself. Our current controlled experiments therefore focus on separating the effects of appraisal supervision and model architecture under comparable training conditions.

Second, the choice of reward is non-trivial for this task. Using final emotion correctness as the reward would primarily optimize emotion classification rather than appraisal reasoning, whereas directly using AEQS as both the RL reward and the evaluation metric could encourage optimization toward the evaluation metric without independently establishing better grounding. A more principled RL formulation would require an independent grounding-aware reward or human-preference signal that evaluates whether the generated appraisal is genuinely supported by the multimodal evidence.

Beyond correct emotion prediction, our aim is to make the intermediate reasoning more interpretable and better aligned with human appraisal of multimodal evidence. Therefore, in the current work, we focus on establishing and validating the Perception → Appraisal → Emotion paradigm through controlled supervision and grounding-oriented evaluation. We view RL with independent grounding-aware or human-preference rewards as an important extension of this framework and will include this direction in our discussion of future work.

Thank you for the thoughtful and detailed response, which addresses most of my concerns. I will therefore maintain my current score, which is already positive.

Reviewer_u65ZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a perception-to-appraisal framework for multimodal emotion understanding.

Existing emotion MLLMs largely rely on direct correlations between surface cues and emotion labels, which can lead to shortcut learning. To address this, the authors model emotion recognition as an explicit trajectory from multimodal perception to cognitive appraisal, covering six dimensions: goal congruence, coping potential, accountability, future expectancy, arousal, and action tendency. The authors construct CogEmo-40K, an instruction-tuning dataset derived from MER2025 using a two-stage automated generation and human-in-the-loop filtering pipeline. They also present CogEmo-MoE, a 3.1B sparse model based on Qwen-1.8B with interleaved MoE blocks trained progressively in three stages. Experiments on a held-out benchmark (CogEmo-Bench) and MERBench evaluate cognitive appraisal evidence quality via a new metric called AEQS alongside standard emotion classification metrics.

Critical strengths
  1. The motivation of grounding multimodal emotion recognition in cognitive appraisal theory is clear and sound. Grounding emotion categories in six structured dimensions provides a much more interpretable reasoning trajectory than unstructured open-ended captions.

  2. The data generation pipeline has structured quality control.

  3. The compact model demonstrates competitive efficiency on standard benchmarks. At 3.1B parameters, CogEmo-MoE matches or outperforms larger open and proprietary models on emotion classification, while achieving strong average performance on MERBench across nine downstream datasets.

Critical weaknesses
  1. The comparison on CogEmo-Bench favors the proposed model due to format differences. CogEmo-MoE is fine-tuned directly on the exact six-dimension chain-of-thought structure, while general baselines generate free-form text parsed by a fixed Qwen3.5-9B extractor. Because missing dimensions receive a penalty score of zero, a large part of the AEQS gap likely comes from simple template matching.

  2. The performance gains from the MoE architecture are small compared to data supervision. The dense baseline achieves an AEQS of 70.48, while the 4-expert MoE reaches 70.60, with the 8-expert configuration dropping to 69.76. These numbers show that the performance gains come almost entirely from the multi-stage training data, weakening the claim of MoE specialization.

  3. The proposed AEQS metric lacks human correlation validation. AEQS relies on cosine similarity from a text embedding model, which tends to reward surface word overlap and fixed phrasing over real reasoning. The paper does not provide human correlation analysis, such as Pearson or Spearman rank correlation against expert psychologist ratings, to support AEQS as a valid benchmark metric.

Questions for authors

Please kindly refer to the weaknesses.

6 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer u65Z

Weakness 1: Is the AEQS advantage mainly caused by six-dimension format matching and train–inference format consistency?

Response. Thank you for raising this important concern about the fairness of CogEmo-Bench. In the original evaluation, all general-purpose MLLMs are evaluated under the same task setting, and their free-form responses are normalized by the same fixed Qwen3.5-9B Appraisal Parser before computing AEQS. This design allows different models to reason in their natural generation styles rather than forcing all models to follow a predefined six-field template.

We agree, however, that CogEmo-MoE may benefit from greater train–inference format consistency, since its Stage-3 supervision is explicitly organized around the six cognitive appraisal dimensions. We therefore conducted two additional analyses.

First, we examined AEQS sensitivity to surface-form changes using the Paraphrased Matched condition. We substantially rewrote matched explanations while preserving their appraisal semantics and final emotion.

Candidate AppraisalAEQS ↑
Matched90.31
Paraphrased Matched83.77
Same-emotion Mismatch65.39
Random Mismatch60.23

AEQS decreases from 90.31 to 83.77 under paraphrasing, but remains substantially higher than 65.39 for Same-emotion Mismatch and 60.23 for Random Mismatch. This suggests that AEQS is influenced by linguistic realization to some extent, but is not dominated by fixed phrasing or template-level matching.

Second, we conducted a schema-controlled evaluation on representative strong baselines. We explicitly provided GPT-5.2 and Qwen3-Omni with the same six appraisal dimensions—Congruence, Coping, Accountability, Future, Arousal, and Action Tendency—while allowing them to reason in natural language and omit dimensions unsupported by the available multimodal evidence.

ModelAEQS ↑Emotion Acc. ↑Macro-F1 ↑
GPT-5.2 (schema-controlled)39.7346.5338.69
Qwen3-Omni-30B-A3B-Instruct (schema-controlled)48.8160.9050.08
CogEmo-MoE70.6071.4357.17

Even with the same appraisal schema explicitly provided, the performance gap remains substantial. CogEmo-MoE achieves 70.60 AEQS, compared with 39.73 for GPT-5.2 and 48.81 for Qwen3-Omni, while also retaining higher emotion Accuracy and Macro-F1. Therefore, simply exposing a general-purpose MLLM to the same six appraisal dimensions is insufficient to reproduce CogEmo-MoE's performance. One plausible interpretation is that current general-purpose MLLMs still have difficulty establishing a reliable mapping from multimodal perceptual evidence to higher-level cognitive appraisal states. In our schema-controlled setting, the baselines were explicitly allowed to omit dimensions that could not be sufficiently supported by the available multimodal evidence, rather than being forced to complete all six dimensions. Under this less restrictive setting, the baselines did not benefit substantially from simply being given the appraisal schema. This suggests that the main challenge is not merely knowing the names or structural definitions of the six appraisal dimensions, but learning how audiovisual and textual evidence should be transformed into cognitively meaningful appraisal representations.

Overall, the paraphrase robustness test and schema-controlled evaluation suggest that the observed advantage is not solely attributable to fixed wording or simply knowing the six appraisal dimensions. Standard emotion Accuracy and Macro-F1 provide a complementary assessment of the final prediction outcome, where CogEmo-MoE also retains an advantage. We will add these analyses to the revised manuscript and clarify both the potential influence of response structure and the intended scope of AEQS.

AuthorsOpenReview ↗

[2/3] Response to Reviewer u65Z

Weakness 2: How much of the gain comes from appraisal supervision rather than the sparse MoE architecture?

Response. We appreciate the reviewer’s focus on disentangling the sources of the observed gains. We agree that, if AEQS is considered alone, the additional gain from the sparse MoE architecture over the Dense counterpart is small: under the same Appraisal + Emotion supervision, Dense reaches 70.48 AEQS, while the 4-expert Top-2 MoE reaches 70.60, and increasing the number of experts to 8 does not further improve AEQS. We therefore agree that the cognitive appraisal capability of CogEmo-MoE should not be primarily attributed to the MoE architecture itself.

To disentangle the effects of appraisal-guided supervision and sparse architecture, we conducted a controlled 2 × 2 ablation. Stage 1 and Stage 2 are fixed, all models start from the same Stage-2 checkpoint, and the source samples, training budget, and hyperparameters are kept identical. We vary only (1) the Stage-3 supervision target (Emotion Only vs. Appraisal + Emotion) and (2) the architecture (Dense vs. 4-expert Top-2 MoE).

Stage-3 SupervisionArchitectureAEQS ↑Emotion Acc. ↑Macro-F1 ↑
Emotion OnlyDense2.1066.7453.60
Emotion Only4E Top-2 MoE1.9868.3553.04
Appraisal + EmotionDense70.4867.2653.27
Appraisal + Emotion4E Top-2 MoE70.6071.4357.17

*Emotion-Only models do not receive explicit appraisal supervision, but they may still occasionally generate limited appraisal-like content from their pretrained capabilities that can be extracted by the Appraisal Parser. As a result, their AEQS scores remain very low, but are not strictly zero.

The results make the attribution clearer. Explicit appraisal supervision is the primary source of the model's cognitive appraisal reasoning capability. Without appraisal supervision, Dense and MoE achieve only 2.10 and 1.98 AEQS, respectively, indicating that little evaluable appraisal content is elicited under the current protocol. Once Appraisal + Emotion supervision is introduced, both architectures reach approximately 70.5 AEQS, confirming that the emergence of explicit appraisal reasoning is mainly driven by the supervision signal rather than by sparsity itself.

At the same time, the sparse architecture provides a different, complementary benefit. Under Emotion-Only supervision, replacing Dense with MoE changes Accuracy from 66.74% to 68.35% (+1.61) and Macro-F1 from 53.60% to 53.04% (−0.56), indicating only a limited and inconsistent gain. In contrast, under Appraisal + Emotion supervision, the same architectural change improves Accuracy from 67.26% to 71.43% (+4.17) and Macro-F1 from 53.27% to 57.17% (+3.90), while AEQS changes only marginally (+0.12).

This distinction is important for how we interpret the model. AEQS primarily reflects the quality of the intermediate cognitive appraisal reasoning, whereas Accuracy and Macro-F1 reflect the final emotion discrimination outcome. Our results suggest that appraisal-guided supervision establishes the Perception → Appraisal reasoning pathway, while sparse conditional computation provides an additional benefit that becomes more visible in the final emotion prediction once this intermediate representation has been learned.

We therefore agree with the reviewer that the original wording may overstate the role of MoE specialization. We do not claim that sparse MoE is the main source of cognitive appraisal ability, nor that increasing the number of experts should monotonically improve performance. Instead, we will revise the paper to position CogEmo-MoE more precisely: the appraisal-guided training paradigm provides the main cognitive reasoning capability, while sparse MoE serves as a complementary architectural enhancement within the proposed Perception → Appraisal → Emotion framework.

Accordingly, we will narrow the MoE-related contribution claim in the revision and emphasize that the broader contribution of this work lies in jointly establishing the cognitive appraisal formulation, cognition-grounded data and evaluation, progressive training pathway, and its architectural realization, rather than attributing the overall improvement to MoE alone.

AuthorsOpenReview ↗

[3/3] Response to Reviewer u65Z

Weakness 3: Does AEQS correlate with human judgments, rather than merely rewarding lexical overlap or fixed phrasing?

Response. This suggestion motivates a direct human validation of AEQS. To examine whether AEQS reflects human-perceived appraisal quality rather than only embedding-level similarity, we conducted a blinded human evaluation on a randomly sampled 20% subset of CogEmo-Bench-Eval (201 source samples). Each source sample corresponds to four candidate explanations, each containing appraisal reasoning across six dimensions, resulting in 804 candidate explanations and up to 4,824 appraisal-dimension-level statements (201 × 4 × 6) to be checked against the underlying multimodal evidence.

Evaluators were shown only the original multimodal input and one candidate appraisal explanation, without access to its experimental condition, AEQS score, or reference source. They rated the extent to which the explanation was supported by the current multimodal evidence on a 1–5 scale. The evaluation involved five evaluators, each with at least two years of research experience in affective computing. Each candidate explanation received one blinded human rating.

ConditionAEQS ↑Human Evidential-Support Rating ↑
Matched91.614.33 ± 0.77
Paraphrased Matched83.934.30 ± 0.83
Same-emotion Mismatch66.042.73 ± 1.14
Random Mismatch59.501.60 ± 0.86

The human scores are reported as the mean ± standard deviation across the 201 candidate explanations within each condition. The human judgments follow the same overall ordering as AEQS. In particular, evaluators assign nearly identical ratings to Matched and Paraphrased Matched explanations, while substantially lower ratings are given to both mismatch conditions. This pattern is consistent with AEQS and provides evidence that the metric is not primarily driven by surface-form overlap.

More importantly, we computed the correlation at the individual-explanation level, rather than over the four condition-level averages. Across all 804 candidate explanations, AEQS shows a strong positive correlation with blinded human judgments:

  • Spearman rank correlation: (ρ=0.703,p<0.001\rho = 0.703, p < 0.001)
  • Pearson correlation: (r=0.729,p<0.001r = 0.729, p < 0.001)

Because the human ratings are ordinal, we treat Spearman correlation as the primary statistic, with Pearson correlation reported as a complementary analysis. These results show that explanations receiving higher AEQS scores are systematically more likely to receive higher human ratings.

We therefore view AEQS as a scalable proxy for large-scale appraisal-oriented evaluation, rather than as a direct replacement for human grounding assessment. The human-correlation results provide empirical validation that the appraisal-reference semantic consistency measured by AEQS is meaningfully aligned with independent human judgments. We will add this analysis to the revised manuscript and clarify both the empirical validity and the limitations of AEQS. We are also continuing to expand the human annotation pool and evaluation coverage to further strengthen this validation.

Comments on the paper

4 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer fJhZ

Weakness 1 & Question 1: Can AEQS distinguish grounded explanations from semantically plausible but sample-mismatched explanations?

Response. We appreciate the opportunity to clarify the intended role of AEQS. Our work introduces cognitive appraisal into large-scale human-centered multimodal emotion understanding and evaluates not only the final emotion category, but also whether the intermediate appraisal reasoning is consistent with a human-verified cognitive reference. We agree with the reviewer that this distinction should be stated more precisely: AEQS directly measures semantic alignment between a generated appraisal and the human-verified appraisal reference, rather than performing audiovisual entailment verification itself. We will therefore describe AEQS more explicitly as an appraisal-reference semantic alignment metric and clarify this scope in the revision.

To directly examine whether AEQS merely rewards recurring appraisal language or emotion-specific templates, we conducted a new grounding stress test. For each test sample, the human-verified appraisal of the current sample was always kept as the reference, while we constructed four types of candidate explanations:

  1. Matched: the appraisal corresponding to the current sample;

  2. Paraphrased Matched: a meaning-preserving paraphrase of the matched appraisal with substantially different wording and sentence structure;

  3. Same-emotion Mismatch: an appraisal transferred from another sample with the same emotion label, but a different event and appraisal content;

  4. Random Mismatch: an appraisal transferred from an unrelated sample.

The same-emotion mismatch is particularly informative because the final emotion category is held fixed. Therefore, if AEQS primarily relied on emotion-specific appraisal templates or recurring phrasing, these explanations should remain close to the matched cases.

Candidate AppraisalAEQS ↑Δ vs. Matched
Matched90.31–
Paraphrased Matched83.77−6.54
Same-emotion Mismatch65.39−24.92
Random Mismatch60.23−30.08

The results show a clear ordering:

Matched > Paraphrased Matched > Same-emotion Mismatch > Random Mismatch.

Importantly, the meaning-preserving paraphrases retain a high AEQS of 83.77, despite substantial changes in surface wording, whereas transferring an appraisal from another sample with the same emotion label reduces AEQS to 65.39, a gap of 18.38 points from the paraphrased matched condition. The score decreases further to 60.23 for randomly mismatched explanations. This indicates that AEQS is not simply rewarding fixed six-dimension phrasing or emotion-category templates; it is sensitive to whether the configuration of appraisal content is consistent with the current instance-specific human reference.

This observation is also consistent with the cognitive-appraisal perspective underlying our framework (Figure 4): emotions are characterized by different configurations across appraisal dimensions, rather than by mutually exclusive values on each individual dimension. Consequently, mismatched explanations need not receive zero scores because different events and emotions can still share partial appraisal characteristics, while differing in their overall appraisal configuration.

Regarding input modalities, Table 2 already reports a V+T comparison: CogEmo-MoE achieves 70.03 AEQS, compared with 56.30 for GPT-5.2 under the same V+T modality setting. Thus, the AEQS advantage is also observed without an additional audio input.

We therefore agree that AEQS should not be interpreted as a standalone audiovisual entailment verifier. Rather, these results support its use as a scalable measure of instance-level appraisal-reference consistency: it remains relatively robust to paraphrasing while substantially penalizing plausible but sample-mismatched appraisal explanations. We will add this stress test and clarify the scope and limitations of AEQS in the revised manuscript. We additionally report standard emotion Accuracy and Macro-F1 as complementary measures of the final prediction outcome; these complement AEQS but do not by themselves verify explanation grounding.

AuthorsOpenReview ↗

[2/3] Response to Reviewer fJhZ

Weakness 1 & Question 2: How does AEQS correlate with blinded human judgments of whether an appraisal explanation is actually supported by the current multimodal evidence?

Response. Thank you for this insightful suggestion. To further validate whether AEQS is aligned with human judgments of evidence grounding, we conducted a new blinded human evaluation on a randomly sampled 20% subset of CogEmo-Bench-Eval, comprising 201 test samples. Given that each source sample corresponds to four candidate explanations and that each candidate contains appraisal reasoning across six dimensions, the evaluation requires careful inspection of the original video, audio, textual content, and the corresponding appraisal statements under a fine-grained evidential-support criterion. The resulting evaluation covers 804 candidate explanations, corresponding to up to 4,824 appraisal-dimension-level statements (201 × 4 × 6) that must be checked against the underlying multimodal evidence.

For each sample, we reused the four candidate appraisal explanations from the grounding stress test: Matched, Paraphrased Matched, Same-emotion Mismatch, and Random Mismatch, resulting in 804 candidate explanations in total. For each candidate, the evaluator was shown only the original multimodal input and the appraisal explanation to be assessed, without access to its experimental condition, AEQS score, or reference source. The evaluator then rated:

“To what extent is this appraisal explanation supported by the multimodal evidence in the current sample?”

on a 1–5 scale, where 1 indicates almost no evidential support and 5 indicates that the explanation is fully supported by the current multimodal evidence.

The evaluation involved five evaluators, each with at least two years of research experience in affective computing. Each candidate explanation received one blinded human rating.

The human ratings exhibit the same overall ordering as AEQS:

ConditionAEQS ↑Human Evidential-Support Rating ↑
Matched91.614.33 ± 0.77
Paraphrased Matched83.934.30 ± 0.83
Same-emotion Mismatch66.042.73 ± 1.14
Random Mismatch59.501.60 ± 0.86

Here, the reported human scores are the mean ± standard deviation across the evaluated candidate explanations within each condition. Notably, human evaluators assign nearly identical scores to the matched and meaning-preserving paraphrased explanations, while substantially lower scores are given to same-emotion and randomly mismatched explanations. This trend closely mirrors the behavior of AEQS.

More importantly, we performed correlation analysis at the individual-explanation level, rather than over the four condition-level averages. Across all 804 candidate explanations, AEQS shows a strong positive correlation with blinded human evidential-support judgments:

  • Spearman rank correlation: (ρ=0.703,p<0.001\rho = 0.703, p < 0.001)
  • Pearson correlation: (r=0.729,p<0.001r = 0.729, p < 0.001)

Because the human ratings are ordinal, we treat Spearman correlation as the primary measure and report Pearson correlation as a complementary analysis. The strong and statistically significant positive correlation indicates that explanations receiving higher AEQS scores also tend to be judged by humans as being better supported by the current multimodal evidence.

We emphasize that this result does not imply that AEQS itself performs direct audiovisual entailment verification. Rather, together with the grounding stress test above, it provides empirical evidence that appraisal-reference semantic consistency captured by AEQS is strongly aligned with independent human judgments of evidential support. We therefore position AEQS more precisely as a scalable proxy for large-scale appraisal-oriented evaluation, while explicitly acknowledging that direct human grounding assessment remains the stronger but considerably more expensive evaluation protocol.

We will add this blinded human evaluation and the corresponding correlation analysis to the revised manuscript, and clarify both the practical role and the limitations of AEQS. We are also continuing to expand the human annotation pool and evaluation coverage to further strengthen this validation.

AuthorsOpenReview ↗

[3/3] Response to Reviewer fJhZ

Weakness 2: Keeping Stages 1 and 2 fixed, can the effects of appraisal supervision and the sparse MoE architecture be separated more explicitly?

Response. This concern motivates a direct comparison between appraisal supervision and sparse architecture. We agree that the effects of appraisal-guided supervision and the sparse MoE architecture should be disentangled more explicitly. To this end, we conducted a controlled 2 × 2 ablation in which Stage 1 and Stage 2 were fixed and all models were initialized from the same Stage-2 checkpoint. We used the same source samples, training budget, and hyperparameters, while varying only two factors in Stage 3: (1) the supervision target (Emotion Only vs. Appraisal + Emotion) and (2) the architecture (Dense vs. 4-expert Top-2 MoE).

Stage-3 SupervisionArchitectureAEQS ↑Emotion Acc. ↑Macro-F1 ↑
Emotion OnlyDense2.1066.7453.60
Emotion Only4E Top-2 MoE1.9868.3553.04
Appraisal + EmotionDense70.4867.2653.27
Appraisal + Emotion4E Top-2 MoE70.6071.4357.17

*Emotion-Only models do not receive explicit appraisal supervision, but they may still occasionally generate limited appraisal-like content from their pretrained capabilities that can be extracted by the Appraisal Parser. As a result, their AEQS scores remain very low, but are not strictly zero.

The results make the attribution clearer. First, explicit appraisal supervision is the primary source of the model's cognitive appraisal reasoning capability. Under Emotion-Only supervision, both Dense and MoE achieve only very low AEQS scores (2.10 and 1.98, respectively), indicating that only limited appraisal-related content can be extracted from their responses. In contrast, introducing Appraisal + Emotion supervision increases AEQS dramatically to 70.48 for Dense and 70.60 for MoE. This confirms that explicit cognitive appraisal reasoning is driven primarily by the supervision signal rather than by the sparse architecture itself.

At the same time, the sparse architecture provides a different, complementary benefit. Under Emotion-Only supervision, replacing Dense with MoE changes Accuracy from 66.74% to 68.35% (+1.61) and Macro-F1 from 53.60% to 53.04% (−0.56), while AEQS remains similarly low (2.10 vs. 1.98), indicating only a limited and inconsistent gain. In contrast, under Appraisal + Emotion supervision, the same architectural change improves Accuracy from 67.26% to 71.43% (+4.17) and Macro-F1 from 53.27% to 57.17% (+3.90), while AEQS changes only marginally from 70.48 to 70.60 (+0.12).

This comparison suggests that the additional architectural gain of sparse MoE is not primarily in producing better appraisal explanations themselves, but is more evident in the final emotion prediction when the model has already learned an explicit perception → appraisal representation. In other words, appraisal-guided supervision establishes the cognitive reasoning pathway, while sparse conditional computation provides a complementary architectural benefit for modeling the mapping from learned appraisal representations to the final emotion decision.

Importantly, we do not interpret these results as evidence that MoE alone creates cognitive appraisal capability, nor do we claim that the present experiment by itself proves a specific internal expert specialization pattern. Rather, the controlled comparison shows that supervision and architecture play different but complementary roles: appraisal supervision is the dominant source of explicit cognitive appraisal reasoning, whereas sparse MoE contributes additional gains that become substantially more pronounced when combined with appraisal-guided learning.

We will revise the manuscript accordingly and narrow our claim about CogEmo-MoE: its contribution is not a fundamentally new MoE architecture, but rather the use of sparse conditional computation as a complementary architectural choice within the proposed Perception → Appraisal → Emotion reasoning paradigm.

Additional update on full-set grounding validation. We have now extended the blinded human grounding evaluation from the initially sampled 20% subset to the entire CogEmo-Bench-Eval set. With four candidate explanations per sample, the expanded evaluation covers 4,032 candidate explanations and up to 24,192 appraisal-dimension-level statements (1,008 × 4 × 6) checked against the corresponding multimodal evidence.

The full-set human evaluation shows the same clear pattern:

ConditionAEQS ↑Human Evidential-Support Rating ↑
Matched90.314.47 ± 0.75
Paraphrased Matched83.774.48 ± 0.73
Same-emotion Mismatch65.392.44 ± 1.12
Random Mismatch60.231.50 ± 0.80

Notably, Matched and Paraphrased Matched explanations receive nearly identical human evidential-support ratings, despite substantial differences in surface wording. In contrast, Same-emotion Mismatch and Random Mismatch explanations receive markedly lower ratings, even when the final emotion category is preserved in the same-emotion condition. This provides stronger full-set evidence that the distinction is driven by instance-specific evidential consistency rather than surface-form similarity or emotion-category agreement alone.

Across all 4,032 candidate explanations, AEQS also remains strongly aligned with blinded human evidential-support judgments:

  • Spearman rank correlation: (ρ=0.740,p<0.001\rho = 0.740, p < 0.001)
  • Pearson correlation: (r=0.771,p<0.001r = 0.771, p < 0.001)

Compared with our initial subset analysis, this complete evaluation substantially broadens the empirical validation and further supports the use of AEQS for measuring instance-level appraisal-reference consistency under multimodal grounding conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.