acceptodds
Under review as a conference paper at ICLR 2027

EmoR1: Neutral-Calibrated Reinforcement Learning for Multimodal Emotion Recognition

Abstract

Multimodal Emotion Recognition (MER) combines linguistic, acoustic, and visual cues. Reinforcement-learning-based multimodal large language models (MLLMs) improve coarse-grained emotion and polarity prediction, yet two reliability issues remain. Neutral samples may be forced into non-neutral labels (neutral under-prediction), while non-neutral samples may collapse into neutral (neutral over-prediction), distorting the boundary in opposite directions. Modality-aware rewards may also credit mentions of audio or visual cues that do not improve prediction. Aggregate evaluations obscure these failures when they do not isolate neutral performance or test modality contributions. We introduce the Multimodal Affective Reasoning-Integrated dataset (MARI), a 103K-sample corpus with step- wise reasoning chains and cross-model auditing, and EmoR1, a 7B model trained on MARI with NH-GRPO. NH-GRPO combines bidirectional neutral-boundary rewards with paired full-modal and text-only score comparisons. We also introduce NH-MERBench for evaluating exact labels, semantic clusters, and polarity under a strict neutral criterion. On seven public datasets, EmoR1 achieves competitive NH-MERBench performance and supports further study of neutral calibration and modality credit.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Emotion Recognition, Multimodal Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DkzPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EmoR1, a reinforcement learning framework for multimodal emotion recognition that addresses neutral-boundary errors and modality credit assignment. The authors introduce NH-GRPO, combining neutral-aware rewards, hierarchical emotion rewards, and a counterfactual modality reward based on full-modal versus text-only predictions. They also construct MARI, a 103K-sample structured reasoning dataset, and NH-MERBench, a neutral-aware evaluation protocol. Experiments across seven benchmarks demonstrate competitive performance and improvements in neutral emotion recognition.

Critical strengths

The paper presents a relatively comprehensive framework encompassing dataset construction, structured supervision, reward design, and evaluation. Experiments cover seven benchmarks and multiple modality settings, with ablations on reward components and training configurations. The reported improvements in Neutral-F1 also suggest that explicitly addressing neutral-class errors is a useful direction.

Critical weaknesses

1. Limited methodological novelty.

The connection between the identified challenges and the proposed solutions remains weak. Despite framing neutral recognition as a calibration problem, NH-GRPO primarily relies on manually designed rewards over discrete labels. In the Neutral-Recovery bucket, the neutral reward is equivalent to 2Rexact−12R_{\mathrm{exact}}-1, providing no fundamentally new supervision beyond exact-match classification. The paper does not explicitly model confidence or decision margins, nor does it evaluate probability calibration or compare against simpler cost-sensitive learning alternatives.

Similarly, MARI mainly introduces structured, model-generated reasoning annotations derived from existing datasets, without supervision specifically targeting neutral ambiguity or modality attribution. Its evidence-grounding reliability is also uncertain: the automatic auditor described in Appendix B receives textual records rather than the original audio-visual inputs, limiting its ability to verify modality-specific claims.

NH-MERBench also substantially overlaps with existing MER-UniBench datasets and uses the same hierarchical compatibility function as the training reward. Overall, the paper does not sufficiently establish the distinct value of these contributions beyond existing structured supervision and task-specific reward engineering.

2. Unreliable Modality Credit.

The proposed modality reward does not reliably reflect genuine audio-visual contributions. A full-modal rollout may outperform a text-only rollout simply because the latter produces a poor prediction due to sampling variation, even when the former relies primarily on textual cues. Such cases can receive positive modality rewards without actual multimodal reasoning, creating a potential reward-hacking shortcut.

More importantly, Appendix C indicates that the text-only score is cached per sample. Since this fixed reference term cancels under GRPO's within-group reward normalization when clipping is inactive, the modality reward effectively reduces to reweighting existing classification rewards rather than providing an independent modality-attribution signal.

These issues undermine the claimed effectiveness of counterfactual modality credit.

Questions for authors

Q1. Counterfactual Modality Reward. How can the authors demonstrate that the proposed reward genuinely encourages audio-visual evidence utilization, rather than benefiting from weak text-only predictions or merely reweighting existing classification rewards? More convincing empirical evidence is needed to establish the independent contribution of this reward to multimodal reasoning.

Q2. Neutral Calibration. How can the authors demonstrate that NH-GRPO genuinely improves neutral-boundary calibration rather than simply enhancing neutral-class prediction through reward engineering? The current evidence primarily relies on Neutral-F1, while Figure 1 compares metrics with different definitions. Stronger empirical evidence is needed to substantiate the calibration claim and establish the advantages over simpler classification-based approaches.

Q3. MARI annotation reliability. How are generated acoustic and visual descriptions verified when the automatic auditor does not directly access the original modalities? Can the authors report quantitative manual verification results and isolate the benefits of structured reasoning and auditing relative to MER-Caption+?

Q4. Missing comparisons with recent omni-modal emotion models. The paper omits relevant recent methods, particularly Omni-Perception Policy Optimization for Multimodal Emotion Reasoning (ICML 2026), which also uses Qwen2.5-Omni and explicitly optimizes modality utilization and evidence faithfulness. Can the authors include comparisons with OPPO and other competitive omni-modal emotion models, as well as a matched vanilla GRPO baseline? The full-modal improvement over Emotion-LLaMA v2 is only 0.62 points on average, making stronger comparisons and standard accuracy/Macro-F1 results important.

Reviewer_4LPVOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes EmoR1, a multimodal emotion recognition framework based on Qwen2.5-Omni-7B. It introduces MARI, a dataset of 103K samples with structured reasoning annotations, and NH-GRPO, a reinforcement learning approach combining neutral-aware, hierarchical, and modality-gain rewards. The framework is evaluated on seven public benchmarks using the proposed NH-MERBench evaluation protocol.

Critical strengths
  1. The experiments cover seven public benchmarks and compare EmoR1 with several recent multimodal large language models.
  2. The paper provides multiple ablation studies examining reward components, training mechanisms, RL dataset size, group size, and reward weights.
  3. The MARI dataset introduces structured multimodal reasoning annotations and combines several existing affective computing datasets.
Critical weaknesses
  1. EmoR1 relies on the existing Qwen2.5-Omni-7B architecture and standard GRPO optimization. The main methodological contribution is the combination of task-specific rewards and sample selection, without introducing a new architecture or optimization algorithm.
  2. Although AffectGPT-R1, R1-Omni, and EMO-R3 are discussed in Related Work, they are absent from the main experimental comparison. The paper also lacks a controlled comparison with standard GRPO using the same backbone, training data, and computational budget.
  3. The main SOTA comparison uses Overall-HScore, which incorporates the same hierarchical compatibility function used during training. A comprehensive baseline comparison using standard classification metrics is missing.
  4. EmoR1 improves the average Overall-HScore over Emotion-LLaMA v2 by only 0.62 percentage points in the full multimodal setting. The paper does not report standard deviations, confidence intervals, or statistical significance tests.
  5. The datasets contain heterogeneous annotations, including continuous sentiment scores, categorical emotions, and multi-label annotations. The paper does not sufficiently explain how the original labels of each dataset are mapped to generated emotion descriptions, canonical labels, polarity, and neutral categories.
  6. MARI relies primarily on Gemini-generated explanations and DeepSeek-based auditing. The paper lacks comprehensive quantitative analysis of annotation accuracy, inter-annotator agreement, rejection rates, and the reliability of the generated evidence, including references to facial Action Units.
  7. The comparison with MER-Caption and MER-Caption+ does not isolate the benefit of stepwise reasoning from differences in dataset composition and size. A controlled comparison between structured and unstructured explanations using identical training samples is missing.
  8. Comparing full-modal and text-only predictions does not establish the individual contributions of audio and visual information. The paper lacks systematic modality ablations, modality-swapping experiments, and evaluation under conflicting multimodal evidence.
  9. Insufficient analysis of error-specific rewards and sample selection. The six RL buckets are introduced, but their sample distributions and individual effects are not adequately examined. Separate evaluations of neutral suppression, neutral recovery, polarity reversal, and fine-grained emotion confusion are missing.
  10. The sensitivity analysis examines a small set of manually selected coefficient variations, without clearly explaining the original coefficient selection procedure or the validation data used for tuning. Interactions between reward components are not investigated.
  11. MARI combines seven existing datasets, but the contribution of each source to the final performance is not evaluated. The effects of dataset imbalance and source-specific annotation characteristics remain unclear.
  12. All experiments use Qwen2.5-Omni-7B, leaving the transferability of NH-GRPO to other multimodal architectures untested. Although CH-SIMS and MER2023 are not included in MARI, the paper does not provide a dedicated analysis of cross-dataset transfer or potential source overlap.
  13. The proposed pipeline involves large-scale annotation generation, independent auditing, supervised fine-tuning, and RL with paired modality comparisons. The additional training cost and inference efficiency relative to standard SFT and GRPO are not systematically evaluated.
  14. The reproducibility and availability of the proposed resources remain unclear. Although the authors state that MARI and NH-MERBench are released, no repository or download links are provided. The availability of the EmoR1 implementation, NH-GRPO training code, model checkpoints, and the selected 6K RL samples is also unspecified.
Questions for authors
  1. What is the main methodological contribution of NH-GRPO beyond combining standard GRPO with task-specific rewards and sample selection? Why are neutral-boundary errors prioritized over other emotion confusions, and how were the reward coefficients selected? The current sensitivity analysis does not fully justify the proposed reward design. How does NH-GRPO compare with standard GRPO and closely related RL methods under matched conditions?
  2. MARI relies on Gemini-generated explanations and DeepSeek-based auditing, followed by manual spot checks. How many samples were manually inspected, what proportion contained incorrect annotations, and how was agreement between human reviewers assessed? Given that some source datasets provide sentiment rather than discrete emotion labels, how was the correctness of generated emotion labels and multimodal evidence, particularly facial Action Units, independently verified?
  3. What are the class distributions and confusion matrices for the evaluated datasets? How does EmoR1 distinguish closely related or potentially ambiguous emotions, such as anger versus fear or joy versus surprise, particularly when surprise can have either positive or negative valence? Does neutral-aware optimization improve these distinctions, or could the reported gains primarily reflect better prediction of a dominant neutral class?
  4. What is the individual contribution of each source dataset to the final performance? Can the authors provide a complete dataset-wise description of the original emotion and sentiment labels, their conversion into the target emotion categories, and the mapping between original annotations, generated descriptions, and evaluation labels? How is consistency maintained across datasets with different annotation schemes, including continuous sentiment and multi-label emotions?
  5. Given the relatively small improvement of 0.62 percentage points over the strongest baseline in the full multimodal setting, are the gains statistically significant across multiple runs? What are the training costs and inference latency compared with the strongest baselines? Will the MARI annotations, NH-MERBench implementation, training code, model checkpoints, and RL samples be publicly released to enable independent reproduction of the results?
  6. What are the individual contributions of audio and visual modalities to EmoR1's performance? Although Table 2 reports Audio+Text, Video+Text, and Audio+Video+Text results, Audio-only and Video-only evaluations are missing. Can the authors provide these results and clarify whether the proposed modality-gain reward improves the effective use of each modality independently?
Reviewer_M4SROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses two reliability problems in RL-trained MLLMs for multimodal emotion recognition (MER): (i) the neutral boundary can be distorted in two opposite directions (neutral under-prediction and over-prediction), and (ii) modality-aware rewards can credit lexical mentions of audio/visual cues that do not actually improve prediction. The authors contribute (a) MARI, a 103K-sample corpus of structured, teacher-generated (gemini-3-flash-preview) and cross-audited (DeepSeek-V3.2) evidence-grounded reasoning traces derived from seven public MER datasets; (b) EmoR1, a Qwen2.5-Omni-7B + LoRA model warm-started on MARI and then trained with NH-GRPO — standard GRPO with bucket-conditioned rewards combining exact-match, a hierarchical compatibility function H (Eq. 2), a bidirectional neutral term R_neu (Eq. 3), and a counterfactual modality term R_cfmodal = clip(S_full − S_text) (Eq. 4–6); and (c) NH-MERBench, an evaluation protocol whose primary metric, Overall-HScore, reuses the same function H. On seven datasets, EmoR1 reports the best average Overall-HScore in all three modality regimes (Table 2), with ablations of reward terms (Table 5), mechanisms (Table 6), reward weights (Table 10), and pool/group sizes (Figure 4).

Critical strengths
  1. Well-diagnosed problem. The neutral-boundary failure mode is real and empirically motivated (Figure 1a: competitive NonNeutral-HScore alongside sharply lower Neutral-F1 across compared MLLMs). Distinguishing under- vs. over-prediction of neutral (Sec. 1) is a useful conceptual decomposition that most MER work glosses over with a single label-level objective.

  2. Counterfactual modality credit is the right idea. Defining modality reward as a score difference between paired full-modal and text-only rollouts (Eq. 4–6) instead of rewarding the presence of words like "tone" or "smile" directly attacks a known reward-hacking channel. This is more principled than the modality-mention rewards used in prior emotion-RL work (R1-Omni, AffectGPT-R1), and the paper is commendably honest that the estimator itself is unchanged GRPO ("The contribution lies in the MER-specific task and reward formulation, not a new policy-gradient estimator", Sec. 2).

  3. Reasonably complete ablation surface. Reward-term ablation (Table 5), mechanism ablation including removal of paired text-only rollouts and bucketing (Table 6), one-dimensional reward-weight sensitivity (Table 10), and RL-pool/group-size sweeps (Figure 4) together cover most of the design space. The finding that removing neutral calibration costs up to 12.5 HScore points on SIMSv2 (Table 6) is informative.

  4. Reproducibility artifacts. Full hyperparameters (Table 9), an implementation-level algorithm (Alg. 1), the actual teacher/audit prompts (App. B), deterministic sanitization rules including label-isolation filtering, and modest compute (4×H100, LoRA 0.37% of parameters) make this unusually easy to replicate for an RL paper.

  5. Useful resource. MARI's three-block schema (modality_extraction / inference_logic / final_judgment) separating observation from inference is a sensible supervision format, and a 103K multi-source corpus with explicit test-split exclusion (App. B, Table 8) is a real community contribution if released as promised.

Critical weaknesses
  1. The headline metric is the training reward (Goodhart circularity). NH-MERBench's primary metric, Overall-HScore, is the mean of h_i = max H(y_i, ŷ) (Sec. 3.4), where H is the same function used as the training reward R_hier = H(y, ŷ) (Sec. 3.3.2), with identical 1.0/0.6/0.25 cluster/polarity constants and identical neutral handling ("the same mapping is used for training and evaluation", Sec. 3.3.2). Every number in Table 2 — the paper's main evidence — therefore measures, for EmoR1, a quantity it was directly RL-optimized to maximize, and for baselines, a quantity they never saw. The gains over baselines are partly guaranteed by construction. The strict metrics that would break the circle (Overall Exact Acc, Macro-F1) are defined but never reported in any results table; only Neutral-F1 and NonNeutral-HScore appear as diagnostics (Table 4), and NonNeutral-HScore still contains H. At minimum the paper needs a table on standard, reward-independent MER metrics (exact accuracy / weighted-F1 under each dataset's native protocol) for EmoR1 and the same baselines.

  2. No vanilla-GRPO control, so the claimed contribution is not isolated. The paper's stated contribution is the reward/bucket formulation, yet no experiment runs "MARI SFT + standard GRPO (R_exact + R_format only, no buckets)" — the single control that would attribute gains to NH-GRPO rather than to RL-on-MARI in general. Table 5's "reduced reward" rows each add one term to R_exact+R_format (69.03 / 66.50 / 68.46 vs. 65.61 without RL) but never show R_exact+R_format alone; we therefore cannot tell how much of the 65.61 → 74.51 jump any plain GRPO run would have obtained for free. Relatedly, the two most comparable RL-based MER methods discussed in Related Work — R1-Omni (Zhao et al., 2025) and AffectGPT-R1 (Lian et al., 2025c) — are absent from every results table. Without these, the claim that NH-GRPO improves over emotion-RL baselines is unsupported.

  3. The margin over the strongest baseline is within plausible noise, and all results appear single-run. In the full audio-visual-text regime (Table 2), EmoR1 averages 70.70 vs. Emotion-LLaMA v2's 70.08 — a 0.62-point gap — and Emotion-LLaMA v2 actually beats EmoR1 on SIMSv2 (74.08 vs. 73.64) and nearly ties on MER2023 (69.22 vs. 69.95) and MER2024 (74.26 vs. 74.51). GRPO training is high-variance; with G=4 rollouts and a 6K pool, run-to-run swings of ±1 point are common. No seeds, no confidence intervals, no significance tests are reported anywhere. Given Weakness 1 (the metric favors the trained model), a 0.62-point average margin is not convincing evidence of superiority over the strongest baseline.

  4. The counterfactual modality signal decays into a stale reference during training. Per Alg. 1 (lines 10, 16–19) and App. C, S_text is cached during RL-pool construction with the initial policy and never refreshed. As RL improves the policy overall, R_cfmodal = clip(S_full − S_text) increasingly conflates general capability gain with modality-specific gain: a sample where the text-only path would also have improved under the current policy still yields positive "modality" credit. The MG bucket membership suffers the same staleness — it is defined by the initial policy's paired gap (Sec. 3.3.1). The paper's central anti-hacking mechanism is therefore only a true counterfactual at step 0. An online text-only branch (or periodic re-caching) is computationally feasible at this scale and its absence is not discussed.

  5. MARI's "evidence grounding" is not verifiable by its own audit. The DeepSeek-V3.2 auditor receives "the transcript, source label, and generated JSON analysis" (App. B) — i.e., no video and no audio. The evidence_fabrication audit category can thus only detect internal inconsistency, not whether claimed AU12 lip-corner raises or "light prosody" actually exist in the media. For a dataset whose raison d'être is grounding predictions in observable evidence, the quality gate cannot check grounding. Furthermore, App. B reports stratified manual spot checks but gives no numbers: sample size, observed error rate, or inter-checker agreement are all missing, so the 103K corpus's actual quality is unquantified.

  6. Over-designed reward machinery with unprincipled constants. The full system has six buckets, five reward terms, and at least seven hand-set constants (H's 1.0/0.6/0.25; R_neu's 0.3 partial credit; S_full's 0.5/0.5 mixing; λ = (1.0, 0.35, 0.4, 0.25, 0.1)). Only the λ's get a sensitivity analysis (Table 10, where the default wins by ≤1.15 points — itself suspiciously convenient as "the default gives the best average"); the H constants, the 0.3, and the 0.5 mixing coefficients are never ablated or justified. The bucket mechanism adds engineering surface (six categories assigned using ground-truth labels and policy errors — the assignment policy is itself underspecified, see Q3) but Table 6's single "w/o bucket" row does not show which buckets carry the benefit. A simpler design (e.g., three buckets, H with only exact/neutral branches) might capture most of the gain; the paper does not rule this out.

  7. Benchmark self-dealing and weak baseline pool. NH-MERBench is introduced by the authors, used as the sole evaluation, and its label resolver is DeepSeek-chat (App. B) — the same model family that audited EmoR1's training data, creating a subtle shared-bias channel favoring EmoR1's DeepSeek-audited output style. Meanwhile many Table 2 baselines (SECap, PandaGPT, Chat-UniVi, Video-LLaVA, OneLLM) are general-purpose models never trained for MER; their inclusion inflates the table's apparent breadth without strengthening the comparison that matters (Sec. 4.2 even acknowledges cross-regime comparisons are invalid, yet the within-regime MER-specific competitor set is thin — e.g., no R1-Omni, no Emotion-LLaMA v1 in AT/VT blocks, no Light-MER outside AVT).

Questions for authors

SEE WEAKNESS

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.