acceptodds
Under review as a conference paper at ICLR 2027

DSPO: Diversity-aware Subjective Policy Optimization for Robust Emotional Reasoning

Abstract

Reinforcement Learning has significantly advanced the complex reasoning capabilities of Multimodal Large Language Models (MLLMs). However, prevailing RL algorithms, such as Group Relative Policy Optimization (GRPO), suffer a severe failure in emotion reasoning tasks. These methods heavily rely on deterministic hard-label supervision and point-wise isolated evaluation, creating a fundamental gap with the inherently subjective and continuously distributed nature of human emotions. Furthermore, unlike explicit physical objects, emotional states are deeply implicit within visual cues. This abstract nature exacerbates visual hallucinations in MLLMs, leading to plausible yet ungrounded emotional evidence. To address these limitations, we propose Diversity-Aware Subjective Policy Optimization (DSPO), a reinforcement learning framework that jointly promotes subjective affective coverage and visual grounding. First, we construct a context-grounded emotional distribution prior in the VAD space by combining the lexical prior of the annotated emotion with image-specific contextual information. Based on this prior, we introduce a Distribution-Aligned Emotional Diversity Reward (DEDR), which measures the leave-one-out marginal contribution of each candidate emotion within a rollout. DEDR rewards candidates whose inclusion brings the predicted affective set closer to the context-grounded prior, thereby preserving plausible subjective interpretations without encouraging unconstrained dispersion. We further develop Counterfactual Visual Intervention Gating (CVIG), which masks the visual region highlighted in the reasoning process and uses the resulting candidate-wise probability changes to reduce the weights of interpretations unsupported by visual evidence. Extensive experiments demonstrate that DSPO achieves state-of-the-art performance across multiple public benchmarks, especially on the cross-domain performance, i.e., improving +10.8% on average cross-domain accuracy than EMO-R3.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Emotional Understanding, Policy Optimization, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4DgyOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DSPO, a GRPO-based RL framework for emotion reasoning in MLLMs. It addresses visual hallucination and sparse hard-label rewards with two modules. The first is CVIG, which masks the model-predicted evidence region to reward visual grounding and weight candidate emotions. The second is DEDR, which rewards alignment between K candidate emotions and a context-grounded VAD prior. Experiments on EmoSet, Emotion6 and WebEmo with Qwen2.5-VL-3B report gains over GRPO, DAPO and EMO-R3, particularly in out-of-domain accuracy, supported by distribution-based metrics and ablations.

Critical strengths

** Consistent out-of-domain gains.** DSPO improves out-of-domain accuracy over GRPO, DAPO and EMO-R3 under both training protocols and both rollout settings (e.g., 59.38 vs. 53.60 for EMO-R3 at G=8). Robustness across datasets is the key challenge in emotion recognition, so this is the main empirical point in the paper's favor. However, its interpretation is limited by the attribution concerns raised in W7 and Q1.

Reasonably thorough ablations at no inference cost. The paper ablates the two modules, the number of candidates, the diversity formulation and the CVIG design choices. In addition, both modules are used only during training, so inference cost is unchanged. This makes the approach practical and offers some insight into which components drive the gains.

Critical weaknesses

Motivation and claims

W1. Key motivating claims lack empirical support. The paper's motivation rests on three claims: that visual hallucination is "exacerbated" in emotion reasoning, that hard-label rewards cause mode collapse, and that GRPO-style methods "suffer a severe failure." None of these is directly supported. No hallucination metric or training-time diversity statistic is reported. The only evidence is a single case (Fig. 4), in which the "hallucinated" dark clouds are actually present in the image. Moreover, Table 1 shows that GRPO substantially outperforms both zero-shot and SFT, which contradicts the claim of severe failure.

W2. Inconsistency between motivation and task formulation. The method is motivated by the subjective and continuous nature of emotions. Yet the model still outputs a single label, evaluation remains hard-matching accuracy, and the accuracy reward dominates the objective (λ_acc = 1.0 vs. λ_div = 0.3).

W3. Insufficient discussion of related work. The paper does not discuss label distribution learning for visual emotion, nor counterfactual or perturbation-based hallucination mitigation (e.g., visual contrastive decoding). Both are closely related to the proposed modules.

Method

W4. CVIG does not verify what it claims to verify. CVIG only checks whether masking the predicted box changes the candidate probabilities, not whether the textual evidence matches the boxed content. Since both the box and the probabilities come from the same policy model, this is not external verification. The paper’s own criticism of EMO-R3 applies equally here.

W5. DEDR performs first-moment matching rather than distribution alignment. The reward compares only the candidate centroid with a prior mean centered on the ground-truth label, and it never considers the spread of the candidates. In Table 5, Center Distance alone (61.69) barely differs from w/o DEDR (61.29), which suggests that prior alignment contributes little.

W6. The two modules are mechanistically in tension. R_cf rewards a single box that all K candidates depend on, whereas DEDR encourages diverse interpretations that often arise from different regions. As a result, CVIG down-weights plausible interpretations grounded elsewhere; “sadness” in Fig. 4 is an example.

Experiments

W7. Unclear whether baselines use the same prompting template. The first row of Table 3 is identical to GRPO (G=8) in Table 1. Consequently, the effect of the 4-step template is either never validated or mixed into the module contributions. Please specify each baseline’s template and report GRPO with and without it.

Questions for authors

Q1. Handling of label spaces

  • Which label set is used for each dataset? For example, is "neutral" removed from Emotion6, and which level of the WebEmo hierarchy is used?
  • When testing on a taxonomy the model was not trained on (e.g., trained on EmoSet's 8 classes, tested on Emotion6's 6 classes), how is the prediction made and evaluated?
  • Could the out-of-domain gains come from the VAD lexicon mapping between label vocabularies, rather than from modeling subjective diversity? Is there a control that can rule this out?

Q2. Prior construction under the Emotion6 training protocol

  • When training on Emotion6, is the proposed lexicon + caption prior replaced by the ground-truth multi-annotator distribution? If so, does this give DSPO extra supervision that the baselines do not have?
  • How is the 6-class annotator distribution converted into the VAD Gaussian parameters (μ*, σ*)?
  • What are the Emotion6-training results when the proposed lexicon + caption prior is used instead?
Reviewer_qR3dOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DSPO for image emotion reasoning. It combines a distribution-aligned diversity reward with counterfactual visual gating under a GRPO framework, encouraging multiple plausible emotional interpretations supported by visual evidence.

Critical strengths

The method addresses two relevant challenges: subjective ambiguity in emotion labels and hallucinated visual evidence.

The evaluation includes comparison with human annotation distributions and ablations examining both proposed components.

Critical weaknesses

DEDR evaluates candidate centroids and their leave-one-out changes, rather than directly matching the full emotion distribution. Opposing but implausible candidates could cancel out and receive favorable rewards. Can the authors verify that the reward encourages individually plausible interpretations?

The evaluation is limited to image emotion recognition, and the reward relies on emotion-specific VAD representations. Could the framework extend to other subjective tasks, such as aesthetic assessment? Clarifying the required adaptations would help establish its broader applicability.

Questions for authors

Refer to weakness.

Reviewer_paHbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DSPO, a GRPO-based framework for visual emotion recognition that aims to accommodate subjective emotional interpretations while improving visual grounding. Each rollout identifies a visual region, generates multiple candidate emotions, and selects a final label. The method constructs an image-conditioned VAD Gaussian prior from lexical statistics and caption-derived affective estimates, then introduces a diversity reward based on candidate-centroid alignment and leave-one-out contributions. A counterfactual visual intervention module masks the selected region and uses changes in candidate probabilities for reward computation and diversity weighting. The distinctive contribution is this combination of affective-set scoring and region-dependent weighting within RL training. Experiments report improved cross-dataset accuracy and better agreement with human annotation distributions, although important mathematical and evaluation issues limit confidence in the proposed explanation for these gains.

Critical strengths
  1. The paper tackles an important problem in emotional reasoning. Emotional interpretations can legitimately vary across observers, while the supporting explanations should remain grounded in the image. The proposed framework offers a concrete way to incorporate both considerations into reinforcement learning, targeting limitations that conventional hard-label accuracy rewards do not adequately capture.
  2. The cross-domain improvements are substantial, and the distribution-based evaluation provides useful evidence beyond classification accuracy. DSPO improves average out-of-domain accuracy over EMO-R3 by 6.66 and 5.78 percentage points under the two rollout configurations, while maintaining comparable in-domain performance.
  3. The experiments include informative ablations of the main design choices. The paper examines the reward components, candidate counts, alternative diversity objectives, and intervention strategies.
Critical weaknesses
  1. DEDR does not sufficiently establish distribution alignment. Equations (14)–(16) combine candidate-centroid alignment with leave-one-out changes in centroid distance, rather than explicitly matching the candidate distribution to the target distribution. The objective can reward increasing dispersion even when the target variance remains fixed. Table 2 supports empirical effectiveness, but does not resolve this property of the objective. The authors should clarify the scope of the distribution-alignment claim and compare against an explicit distribution-matching objective.
  2. CVIG has an unresolved probability-definition issue and the grounding validation is insufficient. If the probabilities in Equation (9) are normalized over the same candidate set, their differences in Equation (10) sum to zero, making RcfR_{\mathrm{cf}} identically zero. Both probability vectors illustrated in Figure 2 sum to one. Candidate-specific gating can still operate, but the separate counterfactual reward and its ablation require clarification. The authors should precisely define how candidate probabilities are calculated and what text is included in the conditioning context.
  3. The proposed reasoning template lacks a controlled ablation. The Cause-grounded Emotional Thinking template introduces localization, psychological reflection, and multi-candidate deliberation, each of which could improve performance independently of the proposed rewards. Table 3 does not clearly state whether the complete template is held fixed across configurations. Consequently, the contributions of prompting and reward design remain unclear.
Questions for authors
  1. How does DEDR avoid rewarding excessive dispersion when the candidate centroid already matches the prior mean?
  2. Does CVIG distinguish human-verified faithful evidence from fabricated evidence, including when masking area and perturbation strategy are controlled?
  3. Is the complete proposed template used in every configuration of Table 3? What improvement does it provide without CVIG and DEDR?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.