acceptodds
Under review as a conference paper at ICLR 2027

EQ-Agent: From Multimodal Emotion Recognition to Emotionally Intelligent Agents

Abstract

Recent foundation models have predominantly emphasized “IQ-like” capabilities—reasoning, mathematics, coding—while emotional intelligence (EQ) remains comparatively underexplored in the pursuit of artificial general intelligence (AGI). Multimodal emotion recognition (MER) has advanced affect perception but stays perception-centric: recognizing an emotion alone does not let an agent understand context, plan support, or respond effectively. We propose EQ-Agent, a framework that extends affective AI from the MER paradigm to a complete EQ loop spanning perception, understanding, planning, and response. EQ-Agent first establishes EQ-MER, a sub-1B perception foundation built on a Perception-to-Cognition (P2C) Bridge architecture and a two-stage training recipe. We then build EQ-Dataset, a 30,664-record multitask resource integrating social reasoning, supportive response, and structured emotional-reasoning supervision to bridge emotion labels and emotionally informed decisions. Initialized from EQ-MER, EQ-Agentic couples multitask supervised fine-tuning with single-reward, EQ-Judge-guided agentic policy optimization to learn compact agentic UNDERSTAND–PLAN–RESPONSE decisions, while EQ-Harness supplies the agentic runtime and deployment support for the model. On nine-dataset MER-UniBench, EQ-MER reaches an 82.22% macro score with state-of-the-art parameter efficiency; on the agentic EQ-AppBench, EQ-Agentic scores 47.25 under a unified judging protocol applied identically to all baselines. Together, these components form a unified pathway from multimodal emotion recognition toward emotionally intelligent agents; we will release the full project upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Emotion Recognition, Affective Computing, Emotionally Intelligent Agents, Multimodal Large Language Models, Agentic Reinforcement Learning, Parameter-Efficient Learning, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_3gWGOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduces EQ-Agent, an end-to-end framework designed to advance affective AI from passive MER to an active, emotionally intelligent decision loop spanning understand, plan, and response. The main contributions include EQ-MER, a 0.9B parameter perception foundation that connects frozen audiovisual feature extractors with a Qwen3-0.6B language model via a perceptual-to-cognitive bridge employing source-preserving cross-attention and dynamic modality routing, alongside EQ-Dataset, a 30,664-record multitask resource consolidating social reasoning, expert counseling replies, and structured emotional reasoning. To complete the agentic architecture, the paper presents EQ-Agentic, an emotional reasoning engine initialized from EQ-MER and trained through Agentic-SFT and single-reward EQ-Judge Agentic-RL, paired with EQ-Harness, a runtime infrastructure enforcing standardized UPR contracts, prompt-level safety alignment, and streaming edge deployment. The authors evaluate these components on EQ-UniBench, a three-axis evaluation matrix covering perception , video-grounded reasoning , and downstream application including closed-set Judgment and open-ended Response modes. In empirical validation, EQ-MER achieves a SOTA macro score of 82.22 on MER-UniBench with aparameter efficiency of 91.36 points/B, while EQ-Agentic reaches an overall score of 47.25 on EQ-AppBench, outperforming the 8.2B AffectGPT baseline.

Critical strengths

The manuscript is well-structured and written with relative clarity. The progression from passive emotion recognition to active UPR agentic loops is logically motivated and supported by the system architecture (Fig.1), and the P2C bridge design (Fig.2). The experimental protocol, ablation studies, and mathematical definitions are are documented thoroughly across the main text and appendices. The paper offers valuable insights into parameter-efficient multimodal fusion and affective agent alignment. The empirical finding that strong perception alone yields near-zero reasoning on complex multi-step video tasks provides an interesting argument for dedicated agentic UPR fine-tuning. The P2C Bridge solves multimodal alignment and fusion for small LLMs by combining source-preserving cross-attention gating with dynamic modality routing, demonstrating that 0.9B models can surpass 8B+ systems such as Emotion-LLaMAv2 in MER. The paper successfully re-conceptualizes affective AI from passive classification to active, context-aware communication.

Critical weaknesses

EQ-Agentic is omitted from the MME-Emotion reasoning benchmark. Furthermore, its advantage over AffectGPT relies on open-ended Response scores, whereas its closed-set Judgment score drops compared to AffectGPT. This suggests that Agentic-RL optimization may degrade verifiable situational judgment skills. The open-ended response subtests as well as the GRPO reward policy utilize Qwen3.8. Because EQ-Agentic is built upon a Qwen3-0.6B backbone, evaluating Qwen-generated outputs using a Qwen-family judge introduces potential risks of model-family preference bias or alignment overfitting, which could skew the narrow margin over AffectGPT. Too many pre-print references.

Questions for authors
  1. Why was EQ-Agentic omitted from the MME-Emotion evaluation in Tab.2? Does the UPR agentic post-training improve upon EQ-MER's 0.62 reasoning step score on video-grounded emotional reasoning items?
  2. Did the single-reward Agentic-RL overfit to open-ended response generation at the expense of closed-set social judgment?Could a multi-reward formulation preserve deterministic judgment accuracy? 3)Have you verified EQ-Agentic's open-ended response scores on EQ-AppBench using non-Qwen judge models such as GPT-4o, Claude 3.5, or Llama-3-70B to ensure the narrow performance lead is robust against intra-family evaluator bias?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3gWG: MME-Emotion Results, RL Effects, and Non-Qwen Judges

We thank the reviewer for recognizing the P2C Bridge, parameter-efficient multimodal fusion, and the progression from emotion recognition to emotionally intelligent agents. We have added public-benchmark results for EQ-Agentic and evaluations by two non-Qwen judges.

Q1: Does EQ-Agentic improve on the 0.62 MME-Emotion reasoning score?

The initial version omitted this result because EQ-Agentic's three-part UPR output required interface validation with MME-Emotion's variable-length step-extraction and scoring pipeline. To preserve the benchmark's evaluation definition, we did not include results before validation and auditing were complete. We have now evaluated the same checkpoint used for the paper's main results:

ModelMME RecognitionMME ReasoningMME CoTMME efficiency (points/B)
EQ-MER35.700.6218.1620.18
EQ-Agentic30.0615.6122.8425.37

Reasoning improves by 14.99 and CoT by 4.68, with the highest parameter efficiency among all seven evaluated models. MME-Emotion is not used in training, showing that the reasoning gains transfer to a public video-based multimodal benchmark. EQ-Agentic also retains 76.09 on MER-UniBench, slightly above the strongest reproduced baseline, Emotion-LLaMAv2 (75.98, 9.3B), preserving strong perception. Full comparison: response to Reviewer fhyG, W1.

Q2: Does single-reward RL overfit to open-ended responses at the expense of closed-set judgment? Could multiple rewards help?

Assessing RL's effect requires comparing the same model before and after RL; the cross-model gap between EQ-Agentic and AffectGPT cannot itself be attributed to RL. Under the paper's main-table protocol:

StageSocialIQAEmoBenchJudgmentResponseOverall
Agentic-SFT62.5931.2546.9232.0939.51
Agentic-RL62.8629.7546.3048.1947.25
RL$-$SFT+0.27−1.50-1.50−0.62-0.62+16.10+7.74

Closed-set social judgment is not neglected: Agentic-SFT includes supervision on SocialIQA's correct options, while Agentic-RL's single holistic EQ reward evaluates understanding, planning, and response across the complete UPR trajectory, returning one overall score from 0 to 10. After RL, Judgment declines by only 0.62, with SocialIQA actually improving by 0.27; Response increases by 16.10 and Overall by 7.74. These results show a substantial response gain and a small judgment decline; they alone do not establish overfitting to open-ended responses. Under the non-Qwen judge Gemini, the SFT-to-RL Response gain remains at least 15.53.

We appreciate the suggestion of a multi-reward formulation and will explore it in future work to better balance response quality and judgment accuracy.

Q3: Have non-Qwen judges verified the response scores?

Yes. We froze the same model outputs, kept the rubric and candidate-text processing consistent, and re-scored them with two judges from different model families, Gemini and Grok:

Non-Qwen judgeEQ-Agentic's Response advantage over AffectGPT
GeminiAt least +8.16
Grok+4.81

Thus, the Response advantage is observed under both non-Qwen judges, rather than only Qwen. For Judgment, an LLM extracts the chosen option without receiving the question or gold answer, followed by deterministic matching to the gold label. For Response, the judge scores only the final non-empty paragraph after special-token removal, without requiring UPR structural fields.

EQ-Agentic slightly exceeds AffectGPT overall (47.25 vs. 47.09), but we do not present this margin as a statistically significant lead. Our main strength is markedly higher parameter efficiency: the 0.9B model reaches approximately 9×9\times the efficiency of the 8.2B AffectGPT, alongside the same model's SFT-to-RL response gains. Cross-judge evidence supports the Response advantage, not a claim that the overall ranking remains ahead under every judge.

Preprint references. We cite preprints to cover recent relevant work comprehensively and avoid omitting important comparisons. Some appear only as supplementary comparisons in the appendix; the main text's core performance comparisons rely on our reproductions under a unified evaluation protocol. For work we cannot reproduce, we label literature-reported scores separately rather than treating them as our reproduced results. We will also check for and prioritize available peer-reviewed versions.

Given that authors resolved my main technical doubts regarding reasoning transfer and evaluator bias and have addressed all of my technical questions and critical weaknesses, I am raising the score to 4.

Thank you very much for your thoughtful feedback and for taking the time to engage in the discussion. We truly appreciate your careful consideration of our responses and are glad that we were able to address your concerns.

Wishing you all the best in your research and future endeavors!

Reviewer_KDYUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EQ-Agent, a framework that extends multimodal emotion recognition toward emotionally intelligent agents. It introduces EQ-MER with a Perception-to-Cognition (P2C) Bridge for lightweight multimodal emotion perception, and EQ-Agentic for structured emotional understanding, planning, and response generation through supervised fine-tuning and GRPO. The paper further integrates existing emotional reasoning and dialogue resources into EQ-Dataset and introduces EQ-UniBench for evaluation. Experimental results demonstrate strong parameter efficiency in emotion recognition and competitive performance on emotional intelligence tasks.

Critical strengths
  1. Reasonable architectural design and experimental analysis. The P2C Bridge combines source-preserving cross-modal interaction, dynamic modality routing, and capacity adaptation. The ablation studies provide evidence for the effectiveness of these components.
  2. Comprehensive system integration. The paper provides an integrated framework covering multimodal perception, structured emotional reasoning, training, evaluation, and deployment.
Critical weaknesses
  1. Insufficient validation of the transition from multimodal perception to agentic emotional intelligence. EQ-Agentic is primarily trained and evaluated on text-based judgment and supportive-response tasks. Consequently, the experiments do not clearly establish how multimodal emotional perception contributes to downstream reasoning, planning, and response generation. Moreover, EQ-Agentic is not evaluated on MME-Emotion, leaving a gap in the claimed capability progression.
  2. Limited methodological novelty of the agentic component. The proposed Understand–Plan–Response (UPR) structure resembles conventional structured reasoning, while the training procedure largely follows existing SFT and GRPO paradigms. The paper does not sufficiently distinguish its agentic formulation from prior emotional reasoning and supportive dialogue methods, nor does it isolate the contribution of explicit planning.
  3. Potential evaluation bias and limited evidence for performance superiority. The same Qwen-based judging framework is used for Agentic-RL rewards and open-ended response evaluation, potentially favoring outputs aligned with the reward model's preferences. Moreover, evaluating complete UPR outputs may introduce structural advantages over baselines generating direct responses. The improvement over AffectGPT on EQ-AppBench (47.25 vs. 47.09) is marginal, and its statistical significance remains unclear.
  4. Unclear checkpoint selection and insufficient attribution of improvements. The appendix describes selecting the highest-scoring checkpoint across nine MER datasets and using a 30% evaluation panel for EQ-AppBench checkpoint screening. It remains unclear whether test instances participate in checkpoint selection, potentially introducing selection bias. Furthermore, the substantial improvement from GRPO (75.44 to 82.22 on MER-UniBench) calls for more controlled comparisons to distinguish architectural contributions from training-strategy gains.
Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer KDYU: Perception Transfer, Explicit Planning, and Evaluation Bias

We thank the reviewer for the positive assessment of the P2C Bridge design and ablations and of our system integration. We added controlled experiments for each concern.

W1: How multimodal emotion perception helps downstream reasoning, planning, and response. Prior human evaluation supports this: ESCoT (ACL 2024) compares reasoning-chain compositions on the same backbone and setup with three professional annotators; adding emotion recognition to "strategy rationale + response" raises coherence/informativeness/empathy from 1.44/1.19/1.40 to 1.63/1.36/1.66, i.e., recognizing emotion first clearly improves the response. Our measurements agree at two levels. First, perception training transfers to agentic ability: with identical data, exposure, and update budget, changing only the starting point, the EQ-MER-initialized model exceeds the pre-perception-initialized one by +4.22 on EQ-AppBench, with EmoBench rising from 11.13 to 30.25. Second, the gains extend to audiovisual reasoning: on MME-Emotion, a video-based multimodal reasoning benchmark never seen in training, EQ-Agentic raises reasoning from EQ-MER's 0.62 to 15.61 and CoT from 18.16 to 22.84, with the highest parameter efficiency among all models, while keeping 76.09 on MER-UniBench, above the strongest reproduced baseline Emotion-LLaMAv2 (75.98, 9.3B). Tables: response to Reviewer fhyG (W1).

W2: Separate validation of explicit planning, and novelty. Following the reviewer's request to validate explicit planning separately, we trained two models from the same EQ-MER with identical data, exposure, update budget, and RL reward, differing only in whether PLAN is output. Understand–Plan–Response beats Understand–Response by +1.44 overall and +3.45 on Response, and also leads at the intermediate checkpoint (+1.25), consistent with ESCoT, where the full chain with strategy rationale outperforms response-only training on all three human ratings. Unlike prior emotional reasoning and support-dialogue methods, our planning is grounded in multimodal emotion perception, a single sub-1B model handles both choice judgment and open-ended response, and a trajectory-level reward optimizes the whole understand–plan–respond chain. Our core contribution is the paradigm shift from emotion recognition to emotionally intelligent agents (response to fhyG, W2).

W3: Evaluation bias, structural advantage, and 0.16. (1) Same-family judge: re-scored by two other-family judges, Gemini and Grok, EQ-Agentic still leads AffectGPT on Response by at least 8.16 and 4.81, and the SFT$\to$RL Response gain under Gemini is at least 15.53, so RL gains are not merely fitting the Qwen reward. Judgment is not judge-scored: an LLM only extracts the option, without the question or answer, and it is deterministically matched to the gold label. (2) Structure: all models receive the same UPR output instruction, and the judge scores only the final non-empty paragraph of the raw output, requiring no structural field. Baselines also produce this structure: 2,523 of AffectGPT's 3,311 judged outputs contain a complete PLAN segment. (3) 0.16: EQ-Agentic slightly exceeds AffectGPT, but we agree this is not a significant lead (our conclusion states it is within run-to-run variance); our main strength is markedly higher parameter efficiency, ∼9×\sim9\times that of the 8.2B AffectGPT.

W4: Checkpoint selection and sources of gains. Test involvement in selection is common in this line of work: the public evaluation code of AffectGPT, Light-MER, and Nano-EmoX implements per-dataset test-best checkpoint selection, and our 30% EQ-AppBench panel likewise informed hyperparameters and where to stop training (clarified in the revised appendix), never training or formal scores. We acknowledge this bias and ask for a consistent standard (response to fhyG, W3). We report all nine MER datasets from one checkpoint per model; even EQ-MER's final GRPO epoch scores 79.99, above every reproduced baseline. The reported Agentic-RL checkpoint is the final checkpoint at the end of training, and all new controls use the final checkpoint without test scores, with conclusions unchanged. For attribution, the paper separates the two factors: the stage table fixes the architecture and shows the GRPO gain (75.44→82.2275.44\to82.22), while the P2C ablation table fixes the full SFT$\to$GRPO schedule and isolates modules (alignment-and-fusion −5.75-5.75; single paths −1.74-1.74 to −3.77-3.77). Moreover, SFT-only EQ-MER (75.44) already matches the 9.3B Emotion-LLaMAv2 (75.98) and exceeds Light-MER and Nano-EmoX, so the architecture is competitive before GRPO.

Reviewer_zJhVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EQ-Agent, an end-to-end framework that aims to extend multimodal emotion recognition toward a full emotional intelligence loop. The framework consists of four main components: (1) EQ-MER, a sub-1B parameter multimodal emotion perception model; (2) EQ-Dataset, a curated 30,664-sample multitask dataset assembled from SocialIQA, CounselChat, ESD-CoT, and ESConv; (3) EQ-Agentic, a model initialized from EQ-MER and further trained via Agentic-SFT and EQ-Judge-guided GRPO to produce structured emotional decisions and responses; (4) EQ-Harness, a runtime infrastructure for unified task protocols, evaluation, and deployment. The authors propose EQ-UniBench, a three-axis evaluation suite with MER-UniBench, MME-Emotion, and EQ-AppBench. They report that EQ-MER achieves 82.22 macro score on MER-UniBench with state-of-the-art parameter efficiency, and that EQ-Agentic reaches 47.25 on EQ-AppBench, slightly exceeding the 8B-scale AffectGPT baseline.

Critical strengths
  1. The paper aligns its three-axis capability hierarchy with established ability models of emotional intelligence from psychology, which gives the work conceptual coherence and positions MER as a foundation rather than an end goal. The full-stack system design covers data, model, training, and deployment, providing a relatively complete technical pipeline.

  2. EQ-MER achieves competitive MER performance at under 1 billion parameters, outperforming similarly sized baselines on the unified benchmark. The deployment validation on an edge device adds practical relevance for resource-constrained scenarios.

  3. The UNDERSTAND–PLAN–RESPONSE output schema makes the model’s emotional reasoning process explicit rather than opaque, which is a desirable property for explainable affective AI and aligns with chain-of-thought paradigms.

  4. The proposed EQ-UniBench attempts to evaluate emotional intelligence beyond simple classification, covering closed-set judgment, reasoning steps, and open-ended response quality, which offers a more holistic view than standard MER benchmarks alone.

Critical weaknesses
  1. The core methodological components of this work are almost entirely incremental combinations of well-established techniques, with no fundamental insight or algorithmic innovation. For example,
  • The P2C Bridge is a standard assembly of cross-modal cross-attention, residual gating, sample-wise modality routing, and bottleneck projection, all of which have been widely used in multimodal LLM literature. No novel fusion mechanism or architectural principle is proposed.
  • The EQ-Agentic training pipeline is the de facto standard alignment recipe for modern LLMs, simply applied to the emotional domain. The UPR structure is a domain-specific variant of chain-of-thought / ReAct prompting, with no new agentic mechanism.

Overall, the work reads as a systematic engineering project that packages existing components into an integrated system, rather than a research contribution that advances the scientific understanding of machine emotional intelligence.

  1. The central performance claims rely heavily on self-constructed benchmarks and self-administered judging, which introduce substantial bias and undermine reliability. The response-axis scoring uses a Qwen3.8 judge model, while the proposed method itself is built on the Qwen3-0.6B backbone. This creates a strong architectural and training-distribution alignment advantage: the proposed model’s output style, format, and token distribution are inherently more familiar to the judge, whereas baselines from other model families are penalized for style mismatch. The paper explicitly acknowledges that format mismatches caused widespread zero scores for baselines during protocol development, yet it does not mitigate this bias.

  2. EQ-Agentic is explicitly fine-tuned on task-specific data from the same distribution as EQ-AppBench, while competing baselines are evaluated out-of-the-box without any task-specific adaptation. This is equivalent to testing a trained student against untrained baselines on the same exam; the performance gap reflects data exposure, not method superiority.

  3. For multiple MER baselines, the reproduced scores are drastically lower than the originally published results, for example, AffectGPT drops from a reported 79.98 to 68.35 in this paper’s evaluation. The authors do not provide a convincing explanation for this ~11-point gap. The combination of depressed baseline scores and elevated self-scores raises serious concerns about evaluation rigor.

  4. The paper does not evaluate EQ-Agentic on the public MME-Emotion reasoning benchmark, only testing EQ-MER (which scores near zero on reasoning, 0.62). By reporting agentic performance exclusively on their own EQ-AppBench, the authors avoid demonstrating that their method improves general emotional reasoning ability as opposed to overfitting to their custom test suite.

  5. EQ-Dataset is built with heavy reliance on proprietary large language models (GPT-6 Astra, Gemini 3.8 Flash) for screening, annotation, and edge-case handling, with only batch-wise sampling-based human review.

  6. The headline result is a 0.16-point difference (47.25 vs. 47.09), which is well within run-to-run variance and measurement noise. The paper itself admits the margin is narrow, yet it frames the result as a meaningful advance.

  7. On cross-dataset generalization (MME-Emotion), the perception model shows near-zero reasoning ability, confirming that strong MER performance does not transfer to deeper emotional understanding, it is a gap the agentic training does not meaningfully close on public benchmarks.

Questions for authors

Refer to Weaknesses part.

1 comment
AuthorsOpenReview ↗

Response to Reviewer zJhV: AffectGPT vs. AffectGPT-R1, MME Results, and Judge Robustness

We thank the reviewer for recognizing our EQ hierarchy, the lightweight EQ-MER, UPR's interpretability, and EQ-UniBench. We first correct a factual point.

Baseline reproduction (AffectGPT). 79.98 belongs to AffectGPT-R1, a separate row in Table 16. AffectGPT's published score is 74.77 and our reproduction is 68.35; these are different works, so there is no "79.98→68.3579.98\to68.35" drop. An important protocol difference for the remaining 6.42-point gap is checkpoint selection: the official evaluation takes the best result per test set across epoch checkpoints before averaging, whereas we use one officially released checkpoint for all nine datasets without per-dataset stitching (response to Reviewer fhyG, W3). Reproduction does not systematically lower baselines: Light-MER gets 75.04 (published 74.61), Emotion-LLaMA 67.30 (64.17). Even vs. the highest published score, EQ-MER's 82.22 exceeds AffectGPT-R1's 79.98.

MME-Emotion and generalization. 0.62 belongs to EQ-MER, which outputs labels without reasoning steps, so near-zero reasoning follows from its task setting. The first version did not report EQ-Agentic on MME, but missing evidence does not establish a lack of improvement. For the same main-result checkpoint, reasoning rises from 0.62 to 15.61 and CoT from 18.16 to 22.84, with the highest parameter efficiency; MER-UniBench is 76.09, still above the strongest reproduced baseline Emotion-LLaMAv2 (75.98, 9.3B). MME is unseen in training, so the gains transfer to a public benchmark. Table: response to fhyG (W1).

Judge preference and format. (1) The Judgment axis is not judge-scored: an LLM only extracts the chosen option, without the question or gold answer, and the result is deterministically matched to the gold label. (2) To avoid format penalties, the Response judge scores the final non-empty paragraph of the raw output without requiring any structural field, so baselines are not zeroed for ignoring UPR. (3) Re-scored by two other-family judges, Gemini and Grok, EQ-Agentic still leads AffectGPT on Response by at least 8.16 and 4.81, so the advantage is not Qwen-specific.

Data exposure and fairness. Training uses only each source's training split; test items are never trained on. To test whether gains come only from exposure, we ran controls with identical data, exposure, and update budget: changing only the starting point, the EQ-MER-initialized model scores +4.22 overall (EmoBench 11.13→30.2511.13\to30.25); changing only whether PLAN is output gives +1.44 (Response +3.45). With exposure fixed, the differences come from method design (table: response to fhyG, W1).

The 0.16 margin. EQ-Agentic slightly exceeds AffectGPT (47.25 vs. 47.09), but we agree this is not a significant lead; our conclusion states it is within run-to-run variance. Our main strength is markedly higher parameter efficiency: 52.50 vs. 5.74 points/B for the 8.2B AffectGPT (∼9×\sim9\times); within the same model, SFT$\to$RL raises Response from 32.09 to 48.19; and the Response advantage holds under cross-family judges.

"Standard assembly." Gating, routing, and bottleneck projection are descriptive names for readability; similar names do not imply equivalent mechanisms. P2C Bridge integrates source preservation, input-dependent modality allocation, and small-model capacity adaptation for multimodal emotion perception. Removing the alignment-and-fusion stage costs 5.75 points, and single-path ablations cost 1.74–3.77. We would gladly compare with any prior work of equivalent structure and computation.

"Simply applying SFT/GRPO." Our reward evaluates the full Understand–Plan–Response trajectory and the consistency among emotional triggers, context, support strategy, and response. Our SFT loss equally weights token-mean objectives across supervision pools, accounting for task complementarity and dataset sizes. Using SFT/GRPO as optimizers does not erase these domain-specific designs. EQ-MER, EQ-Agentic, EQ-Harness, and EQ-Dataset connect perception, understanding, planning, response, supervision, and runtime into one efficient system, shifting from emotion recognition to emotionally intelligent agents. PLAN's role is supported by our control (+1.44) and ESCoT's human evaluation (response to fhyG, W2).

Data quality. Closed-source models assist screening, rule design, and UPR annotation; original human labels, expert replies, and dialogues are retained. Model judgments are not final ground truth. Rule-based filtering precedes blind model auditing; humans adjudicate conflicts individually. Batches are manually sampled with three seeds; systematic issues trigger re-processing. All pilot annotations are human-checked, followed by another three-seed batch audit after rule freezing. Three qualified annotators conduct all human review. We will release all prompts and configs upon acceptance.

Reviewer_fhyGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a EQ loop spanning perception, understanding, planning, and response. EQ-Agent first establishes EQ-MER, a sub-1B perception foundation. The paper also build EQ-Dataset, a 30,664-record multitask resource. EQ-Agentic use multitask supervised fine-tune and RL learning. EQ-Harness supplies the agentic runtime and deployment support. On nine-dataset MER-UniBench, EQ-MER achieves SOTA parameter efficiency. On the agentic EQ-AppBench, EQ-Agentic achieve comparable performance with 8B-scale baseline AffectGPT.

Critical strengths
  1. The sub-1B perception foundation EQ-MER achieve a new SOTA on MER-UniBench.
  2. The P2B Bridge is specially designed and is validated through ablation study.
Critical weaknesses
  1. EQ-MER performs good at MER-UniBench, but there are no report on MER-UniBench and MME-Emotion for EQ-Agentic.
  2. The novelty of proposed method is limited. Agentic part is supervised learning and traditional GRPO.
  3. As shown in Section B.4, the subset of test set is used for model selection.
Questions for authors

See the weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fhyG: EQ-Agentic MER/MME Results, Novelty, and Checkpoint Selection

We thank the reviewer for the positive assessment of EQ-MER and the P2C Bridge.

W1: EQ-Agentic on MER-UniBench and MME-Emotion. These were omitted initially because EQ-Agentic's fixed three-part UPR output needs adaptation to MME-Emotion's step-extraction and step-wise judging protocol; to respect the benchmark's definition, we did not report results before this was validated and audited. Results for the same main-result checkpoint:

ModelParamsMER-UniBenchOV-MERD+MME RecMME ReaMME CoTMME score/B
EQ-MER0.9B82.2261.0835.700.6218.1620.18
EQ-Agentic0.9B76.0963.4130.0615.6122.8425.37
Emotion-LLaMAv29.3B75.9862.1840.1944.6742.434.56
AffectGPT8.2B68.3555.6726.2224.0125.113.06

After post-training, EQ-Agentic still slightly exceeds the strongest reproduced baseline, Emotion-LLaMAv2, with ∼1/10\sim1/10 of its parameters, and improves the only open-vocabulary task, OV-MERD+ (61.08→63.4161.08\to63.41). The closed-set decline is consistent with a shift toward counseling-style wording (canonical emotion-word rate 93.4%→77.4%93.4\%\to77.4\%). On MME-Emotion, reasoning rises by 14.99 and CoT by 4.68 over EQ-MER, with the highest parameter efficiency.

Two controlled comparisons (identical data, exposure, and update budget; final checkpoint; all arms re-evaluated under one shared protocol, Appendix B.1):

ControlSettingEmoBenchJudgmentResponseOverall
Init.Pre-perception init.11.1336.5049.7843.14
EQ-MER init.30.2546.5548.1647.36 (+4.22)
PlanningUnderstand–Response31.7546.8837.4642.17
Understand–Plan–Response30.3846.3040.9143.61 (+1.44)

Emotion-perception pre-training thus transfers to downstream agentic tasks. All results are now in the revision (Table 2; Appendix B).

W2: Novelty. We agree GRPO is standard and do not claim it. Our core contribution is not a new agent algorithm but a paradigm shift from emotion recognition that outputs labels to an Emotionally Intelligent Agent that understands, plans, and responds, realized end-to-end under 1B parameters. Key designs: (1) the P2C Bridge, validated by the ablations the reviewer acknowledged; (2) a trajectory-level reward: EQ-Judge scores the full Understand–Plan–Response chain holistically, so optimization covers accurate understanding, appropriate strategy, and a consistent response, not only the final reply; (3) multi-task SFT loss and data mixing: equal-weight pools for social judgment, supportive response, and structured emotional reasoning with token-averaged loss, so long dialogues do not dominate gradients; (4) EQ-Dataset and EQ-Harness for unified supervision and interfaces.

Evidence for explicit planning: first, in the control above the RL reward is identical and only the PLAN field differs; UPR is higher by 1.44 overall and 3.45 on Response, and also leads at the intermediate checkpoint (+1.25). Second, in independent human evaluation, ESCoT (ACL 2024) compares reasoning-chain compositions on the same LLaMA2-7B-Chat backbone and setup; three professional annotators rated 50 shuffled responses (0–2). The full chain (emotion, cause, appraisal, strategy rationale) scored 1.65/1.51/1.71 in coherence/informativeness/empathy vs. 1.63/1.32/1.42 for response-only, with 85% strategy consistency. In ESConv (ACL 2021) human interaction tests, a model that predicts a support strategy before responding was preferred over one without strategy (51 wins, 34 losses, 15 ties). Together, these show that explicit understanding and strategy planning improve support responses. Explicit PLAN also makes each chosen strategy and its rationale auditable, aiding error localization and integration with tool calls, strategy constraints, or human intervention.

W3: Checkpoint selection. Test involvement in selection is common in this line of work: on MER-UniBench, the public evaluation code of AffectGPT, Light-MER, and Nano-EmoX selects per-dataset test-best checkpoints before averaging, and our frozen 30% EQ-AppBench test panel likewise informed hyperparameters and where to stop training (clarified in the revised appendix). We acknowledge this selection bias and ask for consistent standards for our reporting and the baselines. Our MER table uses one checkpoint per model across all nine datasets, without per-dataset stitching; even EQ-MER's final GRPO epoch scores 79.99, above every reproduced baseline (best 75.98).

Panel scores never enter gradient training or formal results. The reported Agentic-RL checkpoint is the final training checkpoint, and all new controls fix the final checkpoint without test scores; both gains hold. EQ-Dataset has train/validation/test splits; we will extend validation-based selection to MER-UniBench and other benchmarks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers and the AC. We have uploaded a revised PDF; the main changes are summarized below.

New results

  • EQ-Agentic on MER-UniBench and MME-Emotion (Table 2, Figure 3, Appendices B.3 and C.3): with the same checkpoint used on all three axes, EQ-Agentic scores 76.09 on MER-UniBench and 30.06/15.61/22.84 on MME-Emotion recognition/reasoning/CoT, the highest MME parameter efficiency (25.37 points/B).
  • Controlled agentic comparisons (Appendix B.1, Tables 11–12): with matched data, exposure, and update budget, EQ-MER initialization improves the overall EQ-AppBench score by 4.22 over pre-perception initialization, and explicit PLAN improves it by 1.44 (Response +3.45) under an identical reply-only RL reward. Prior human evaluations of explicit planning (ESCoT, ESConv) are summarized alongside.
  • Independent judges (Appendix B.2, Tables 14–15): Gemini and Grok re-score frozen outputs on both axes. EQ-Agentic's Response advantage over AffectGPT remains at least 8.16 and 4.81 points, and its parameter efficiency stays 8.5–9.1$\times$ that of AffectGPT.
  • Parameter efficiency (EPE_P, points per billion parameters), previously reported for MER-UniBench, is extended to MME-Emotion and EQ-AppBench in Table 2.

Clarifications

  • Evaluation protocol (Section 4, Appendix C): Judgment options are extracted by an LLM that sees neither the question nor the gold answer and are then matched deterministically; the Response judge scores the final non-empty paragraph of the raw output; failure handling for MME-Emotion and EQ-AppBench is stated explicitly.
  • Checkpoint selection (Appendices C.2 and C.4): the role of the 30% EQ-AppBench development panel, the single-checkpoint MER reporting, and the per-dataset test-best selection in public baseline evaluation code are described explicitly; all GRPO epochs of EQ-MER exceed every reproduced baseline.
  • The AffectGPT checkpoint description is corrected to match its official release.

Organization

  • Appendix B now collects the additional experiments and complete results, and Appendix C the evaluation protocols. The Judgment/Response columns of Table 2 are moved to Table 19. Wording was edited for readability; no previously reported number was changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.