acceptodds
Under review as a conference paper at ICLR 2027

EmoCARE: Empowering Visual Emotion Comprehension via Subjective Adaptation and Canonical Optimization

Abstract

Visual Emotion Comprehension (VEC) aims to infer the emotional responses that visual stimuli may evoke in human observers, which requires reasoning beyond visual content to its affective implications. Although chain-of-thought (CoT) reasoning has shown promise for complex reasoning in multimodal tasks, its application to VEC remains challenging, as existing multimodal large language models (MLLMs) often generate unreliable affective reasoning chains. In this work, we systematically investigate CoT reasoning for VEC and reveal that the major bottleneck lies in the quality of self-generated affective reasoning rather than the CoT paradigm itself. Through fine-grained error analysis, we identify four predominant reasoning failure modes and further highlight the challenge posed by subjective emotion ambiguity. To mitigate this, we introduce VECBench-Subjective, a benchmark with subjective emotion annotations that captures the inherent diversity of human affective judgments. Building on these findings, we propose EmoCARE, which leverages Canonical Affective Rubric rEfinement to enhance the quality of affective reasoning. By transforming fine-grained reasoning errors into instance-specific corrective supervision, EmoCARE enables MLLMs to produce more faithful affective reasoning and achieve improved emotion prediction. Extensive experiments on both VECBench and VECBench-Subjective demonstrate that EmoCARE consistently achieves state-of-the-art performance across multiple MLLMs, validating its effectiveness for both conventional and subjective VEC.

Primary area:
foundation or frontier models, including LLMs
Keywords:
CV: Language, Vision & Multi-modal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wfZVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why MLLMs underperform when using CoT for Visual Emotion Comprehension, identifying low-quality self-generated affective reasoning as the main bottleneck, which are characterized by four major failure modes. To address this, the authors construct VECBench-Subjective, a multi-label benchmark based on multi-model voting and human verification, and propose EmoCARE, which converts fine-grained reasoning failures into instance-specific affective rubrics used as process-level rewards in GRPO. Experiments on conventional and subjectivity-aware VEC benchmarks show that EmoCARE improves affective reasoning and emotion prediction, achieving state-of-the-art or highly competitive performance across diverse MLLMs.

Critical strengths

1.The study is well-motivated, beginning with a systematic analysis of how intermediate reasoning quality affects visual emotion comprehension. By clearly categorizing the failure modes of self-generated CoT into four distinct error types, the authors provide a solid empirical foundation that naturally justifies the design of their subsequent methodology.

2.The empirical evaluation is extensive and spans a wide range of benchmarks. The proposed model not only achieves strong in-domain performance but also demonstrates impressive OOD generalization, offering a high-quality and practical solution for advancing MLLM-based image-evoked emotion comprehension.

Critical weaknesses

1.The proposed pipeline is computationally expensive and may be difficult to scale. For every sample in the RL stage, multiple large models (up to 38B) and proprietary APIs (GPT-5) are needed offline to generate candidate rubrics, collect failure cases, and match errors, while Qwen3-32B is continuously queried online during GRPO to score reasoning steps. Moreover, this reasoning supervision heavily relies on an automated LLM-as-a-Judge paradigm without any human verification on a sampled subset, which weakens both the reliability of the reward signals and the technical novelty of the framework.

2.A key motivation of this work is the one-to-many subjective nature of human emotions. However, this subjectivity is only considered during evaluation via VECBench-Subjective, whereas the training stage remains strictly single-label. Specifically, the accuracy reward in Eq. (2) still penalizes any prediction that deviates from the single original ground-truth label, even if the predicted emotion is subjectively valid. Failing to incorporate multi-label or subjectivity-aware supervision into the training objective creates a clear disconnect between the motivation and the method.

Questions for authors

1.In the model voting stage of VECBench-Subjective, Qwen3.6-27B is used to generate candidate labels, while the primary backbone (Qwen3-VL-8B) and the online judge (Qwen3-32B) belong to the same Qwen model family. Could this introduce a potential model-family bias that favors Qwen-based models on VECBench-Subjective?

2.On VECBench-Subjective, the Hit Score metric counts a single-label prediction as correct as long as it matches any label in the ground-truth set. However, for samples with multiple valid labels (e.g., ≥3\ge 3 labels), the random-guess hit rate increases substantially. Why not directly prompt the model to predict a multi-label set or distribution and evaluate it using set-based metrics such as Macro-F1 or IoU?

3.Since both the offline rubric refinement and online GRPO reward scoring rely entirely on LLMs, could the authors provide a small-scale human verification study to assess the quality of the synthesized rubrics and the agreement between Qwen3-32B's reward scores and human judgments?

4.Apart from the error-mode breakdown in Figure 2(a) and case studies in Appendix D, could you provide a more direct quantitative evaluation of the intermediate reasoning quality itself, such as a small-scale study comparing rubric agreement or human preference on the generated reasoning chains?

5.It would be helpful if the authors could further review the manuscript formatting and ensure that the hyperparameters are reported consistently and clearly. In particular, please fix the citation formatting throughout the text (where \citet is frequently used instead of \citep) and specify the exact values of missing hyperparameters (e.g., λrub\lambda_{rub} in Eq. 5 and β\beta in Eq. 7) to ensure full reproducibility.

Reviewer_2xM2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses two challenges in visual emotion comprehension: single emotion labels often fail to capture the subjectivity of human responses, and the reasoning generated by multimodal models can be flawed and does not consistently improve emotion prediction. The task is to predict the emotions an image may evoke in a viewer and provide a plausible explanation grounded in its visual content and context, while recognizing that multiple emotional responses may be valid. The authors introduce VECBench-Subjective, which combines candidate labels from multiple models, voting, and human verification to construct sets of acceptable emotions. Predictions are evaluated using HIT, which measures whether the predicted emotion belongs to the accepted set. They also propose EmoCARE, which generates fine-grained evaluation rubrics around four types of issues: visual perception errors, emotional interpretation errors, inconsistencies between reasoning and answers, and subjective ambiguity. These rubrics are refined using actual model failures and weighted by their selection frequency, then incorporated into a training pipeline combining supervised fine-tuning with GRPO, using rewards for answer correctness, output format, and rubric satisfaction. Built on Qwen3-VL-8B-Instruct, EmoCARE achieves an average accuracy of 55.38% across four in-domain fine-grained tasks on the original benchmark, outperforming EmoCaliber by 1.40 percentage points. Across all nine original benchmark tasks, it reaches 61.88%, a 3.20-point improvement over supervised fine-tuning alone. On the seven tasks in VECBench-Subjective, it achieves an average HIT of 67.36%, improving on the same baseline by 1.90 points. Validation includes comparisons with general-purpose and emotion-focused models, in-domain and out-of-domain evaluations, ablations of training strategies and rubric construction, and a human preference study of the label sets.

Critical strengths

The paper addresses an important limitation of visual emotion evaluation: the same image can evoke several reasonable emotional responses, while single-label benchmarks may treat valid alternatives as errors. VECBench-Subjective takes this subjectivity seriously by combining multi-model proposals and voting with independent review by three annotators. A study involving five volunteers and 200 images further shows a preference for the revised label sets.

The experimental evaluation also provides useful support for the proposed approach. The authors connect concrete reasoning failures to instance-specific evaluation rubrics and examine their effectiveness across conventional and subjectivity-aware benchmarks, including both in-domain and out-of-domain tasks. Comparisons among SFT, GRPO, and different rubric construction strategies help clarify the value of the training design. Relative to SFT alone, EmoCARE improves average accuracy by 3.20 percentage points across the nine original benchmark tasks and average HIT by 1.90 points across the seven subjective benchmark tasks.

Critical weaknesses

The training objective is not clearly aligned with the paper’s goal of accommodating subjective emotional responses. The appendix defines M4 as cases where the reasoning is coherent and the answer is reasonable but differs from the reference label because of subjectivity. However, the accuracy reward still requires matching a single label, and some rubrics explicitly favor the reference emotion over a plausible alternative. A reduction in M4 could therefore reflect greater conformity to the reference labels rather than a better ability to recognize and preserve valid emotional interpretations.

The ablations also leave the source of CARE’s gains insufficiently resolved. The Full Rubric and Part Rubric comparisons provide useful evidence that rubric construction matters, but they do not separately isolate the effects of error-guided selection, rubric count, frequency-based weighting, and additional teacher supervision. The full method improves average HIT over Part Rubric by 1.34 percentage points, yet no repeated runs or confidence intervals are reported to establish the stability of this difference. It therefore remains unclear how much of the improvement can be attributed to the proposed refinement mechanism.

Questions for authors

Could you clarify how the annotation instructions and rubrics distinguish emotions experienced by the viewer from those expressed by people in the image? The paper defines VEC in terms of viewer responses, but some prompts and examples focus on a depicted person’s expression, intentions, or emotional state. These can inform the viewer’s response without determining it. How are cases handled when the two differ?

How were the three annotators’ judgments combined into the final emotion sets, particularly when they disagreed about adding or removing a label? Could you provide any available agreement statistics or analysis of these disagreements? Given the subjectivity of the task, high agreement should not be the sole criterion for quality; the more important issue is how the procedure distinguishes legitimate differences in emotional response from annotation errors.

What evidence beyond classification scores and qualitative examples supports the claim that EmoCARE improves reasoning quality? The reported human study evaluates annotation schemes rather than model-generated explanations. If an independent, blinded evaluation of model outputs has been conducted, could you report its protocol and results, particularly for visual grounding, emotional plausibility, and consistency between the explanation and final answer? If such evidence is unavailable, could you clarify the intended scope of the reasoning-quality claim?

How does the training objective preserve reasonable alternative interpretations among samples classified as M4, given its reliance on a single-label accuracy reward? As a targeted validation request, could you compare the current objective with variants that exclude M4 from corrective supervision or accept a human-validated set of emotion labels? Reporting both reference-label accuracy and the acceptance of reasonable alternatives would help distinguish improved subjective emotion understanding from increased conformity to the reference labels.

The performance claim should be limited to the relevant aggregate results. Although EmoCARE achieves a higher average ID VER accuracy than EmoCaliber (55.38% vs. 53.98%), its WebEmo-25 accuracy is lower (27.00% vs. 28.00%). The statement that it achieves the best performance on all ID tasks should therefore be revised.

Reviewer_8UBdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual emotion comprehension, focusing on unreliable generated reasoning and ambiguity in image-evoked emotions. It introduces VECBench-Subjective, which extends existing single-label benchmarks using multi-model candidate generation, voting, and verification by three annotators. It also proposes EmoCARE: candidate rubrics are generated from reference reasoning and an error taxonomy, selected against diagnostic model failures, and weighted to provide an additional reward during GRPO after supervised fine-tuning. Experiments use Qwen3-VL-8B as the trained backbone and evaluate conventional and subjectivity-aware emotion prediction. EmoCARE improves average in-domain emotion-recognition accuracy from EmoCaliber's 53.98% to 55.38% and achieves a 67.36% mean hit score on VECBench-Subjective.

Critical strengths
  1. The improvement is supported by conventional evaluation as well as the new benchmark. Tables 3 and 4 report improvements over EmoCaliber in average ID emotion recognition, ID sentiment analysis, and OOD emotion recognition.

  2. The paper addresses a meaningful mismatch between subjective emotion judgments and single-label evaluation. The benchmark construction includes human review rather than relying exclusively on model consensus.

Critical weaknesses
  1. The evidence for the subjective benchmark's validity does not sufficiently separate genuine alternative interpretations from greater label permissiveness. Section 3 does not specify how the three annotators' differing label sets are combined, their agreement, or how often they add and remove model-proposed labels.

  2. The distinctive rubric refinement mechanism is insufficiently specified and isolated experimentally. Tables 6 and 7 provide useful Full Rubric and Part Rubric ablations, but they do not separate failure-based selection from rubric count and frequency weighting.

Questions for authors
  1. How are disagreements among the three annotators resolved, and what agreement and label-edit statistics were obtained?

  2. Please specify how instance-level candidate rubrics become the final per-prompt rubric set, how frequencies determine weights, the value of the rubric reward coefficient, and the reward judge's exact inputs and prompt.

  3. Are results available for uniform weighting of the selected rubrics and for a random subset matched to the selected rubric count, using the same training setup?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.