- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MERRY, a semantic-level evaluation framework for assessing emotional and role consistency in multimodal role-playing agents. It introduces metrics for cross-modal emotional consistency and emotional discrepancy, along with a bidirectional evidence-finding LLM-as-a-Judge protocol. The paper further construct MERRY-Data, and evaluate multiple multimodal models to identify limitations in emotional expression and role consistency. Experiments under different datasets show gains over baselines.
Critical strengths
1.The paper addresses an underexplored problem in multimodal role-playing evaluation by focusing on semantic consistency before modality synthesis. 2.MERRY introduces several metrics and an evidence-based evaluation protocol, providing a more fine-grained perspective on emotional and role consistency. 3.The newly constructed MERRY-Data contains diverse multimodal semantic annotations, and experiments across multiple models provide useful insights into current limitations.
Critical weaknesses
1.The validity of the proposed emotional consistency metrics needs further justification. High cross-modal agreement does not necessarily indicate accurate emotional expression, especially when different modalities intentionally convey contrasting emotions. 2.The evaluation relies heavily on LLM-based judgments. Although human agreement studies are provided, they involve only 100 samples per study, which may be insufficient to validate the reliability of the proposed metrics across diverse emotional contexts. 3.The proposed metrics may favor overly consistent or simplified emotional descriptions. For example, MERRY fine-tuning increases CEC substantially from 0.586 to 0.934, while MEC improves only slightly from 0.264 to 0.291, raising concerns about the relationship between cross-modal consistency and emotional correctness. 4.The generalizability of the benchmark is limited. MERRY-Data is derived from Chinese TV-series dialogues with a relatively small number of test roles, leaving its applicability to broader multilingual and real-world interactive scenarios unclear. 5.Missing references about consistency remaining emotional and role playingb system: [1]Ronghao Lin, Shuai Shen, Weipeng Hu, Qiaolin He, Aolin Xiong, Li Huang, Haifeng Hu, and Yap-peng Tan. 2025. E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Model. In Proceedings of the 33rd ACM International Conference on Multimedia (MM '25). Association for Computing Machinery, New York, NY, USA, 14006–14013. [2]Yirui QI, Xiaoming Zhang, Ruilin Zeng, Mengyao Liu, Ziyi Zhou, Dezhuang Miao, Bingyu Yan, and Zhenyu Guan. 2026. Beyond Static Persona Consistency: Dynamic Persona Coherence in LLM Role-Playing. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28942–28956, San Diego, California, United States. Association for Computational Linguistics.
Questions for authors
1.Could the authors report the correlation between the proposed CEC/ED metrics and direct human judgments of overall multimodal emotional coherence? 2.What is the impact of excluding the fused emotion channel when computing CEC, given that this channel already incorporates information from the other modalities? 3.How sensitive are the evaluation results to different combinations of ERC experts or LLM judges? 4.Can the authors provide an analysis of whether the proposed metrics are affected by the length or level of detail of generated semantic descriptions? 5.Could the authors evaluate more challenging semantic errors, such as subtle emotion substitutions, rather than only randomly mismatched semantic descriptions? 6.Can the authors provide more examples of cases where the proposed metrics disagree with human judgments and discuss the reasons? 7.What is the computational cost of evaluating the complete benchmark, considering the multiple LLM-based emotion recognition and judgment calls? 8.Could the authors further analyze performance differences between frequent and rare emotion categories to assess whether the benchmark captures fine-grained emotional understanding?
6 comments
W3: CEC may favor simplified emotional descriptions? & Q4: how metrics are affected by length
Response: Thank you for raising this question! With regard to the importance and validity of CEC , we have already responded in sections W1 and Q1.
Regarding the effects of fine-tuning, we observed that improvements across different metrics are not uniform following straightforward fine-tuning: given the same data volume and computational budget, the gains for MEC are notably smaller than those for CEC. This may be due to a disparity in the difficulty levels of the two metrics. Consequently, as noted in the final sentence of our analysis in Section 5.2, more targeted training approaches—such as Reinforcement Learning (RL)—may be necessary and useful in the future.
To examine length sensitivity, we additionally analyzed all nine zero-shot models in Table 4. For each model separately, we divided responses into 20 bins based on the total non-whitespace character count. Within each bin, we recomputed CEC and ED, and then calculated Pearson’s between each metric and the actual mean response length across the 20 bins:
| Model | CEC lower | CEC upper | ED fusion | ED speech | ED facial | ED body |
|---|---|---|---|---|---|---|
| OmniCharacter-7B | 0.575 | 0.673 | −0.011 | 0.840 | −0.086 | 0.166 |
| Doubao-1.5-Vision | 0.764 | 0.564 | 0.537 | 0.438 | 0.360 | 0.429 |
| GPT-5-Chat | 0.577 | 0.484 | 0.572 | 0.035 | 0.315 | 0.383 |
| Gemini-2.5-Pro | 0.640 | 0.523 | 0.475 | 0.349 | 0.434 | 0.184 |
| Qwen3-Omni-30B-A3B | 0.919 | 0.914 | −0.680 | −0.788 | −0.807 | −0.805 |
| MiniCPM-o-4.5 | 0.897 | 0.892 | −0.825 | −0.779 | −0.790 | −0.689 |
| MiniCPM-o-2.6 | 0.917 | 0.907 | −0.655 | −0.813 | −0.791 | −0.565 |
| Qwen2.5-Omni-7B | 0.900 | 0.887 | −0.461 | −0.314 | −0.319 | −0.326 |
| Phi-4-Multimodal-5.6B | 0.896 | 0.856 | −0.729 | −0.235 | −0.267 | −0.113 |
Results shows that CEC is positively associated with length for evaluated model, while ED shows model-dependent directions. This indicates that high CEC and lower ED are not solely limited to simple, brief descriptions. Notably, these correlations do not simply imply that CEC has a stronger preference for longer responses, as the specific CEC values still vary across different models.
Q2: CEC without fusion
Response: We include the fusion channel so that CEC captures both agreement among independently inferred modality-specific emotions and their alignment with the emotional interpretation of the complete response. The fusion channel serves as a holistic reference derived from all semantic fields, including the textual content. This fused interpretation is also used in computing other emotional evaluation metrics.
Computing CEC from only the three modality-specific channels measures their mutual agreement but does not directly assess alignment with the complete response. It can therefore overlook cases where the three channels agree with one another but conflict with the joint interpretation, without a coherent explanation for that discrepancy.
Our supplementary human-agreement study provides empirical support for this design choice. As shown in Table 13, excluding the fusion channel reduces agreement between CEC and direct human judgments. This finding supports retaining the holistic reference when assessing cross-modal emotional consistency.
Q6: why disagree with human judgments
Response: Thank you for this suggestion! We have added a targeted human audit of RC evidence and score aggregation in Appendix G. Critical evidence was the most frequently endorsed reason for adjusting individual-judge scores (61/82 groups, 74.39%). These cases highlight potential limitations of evidence counting in capturing the relative importance of supporting and conflicting evidence. But results also found that the presence of critical evidence alone did not determine whether an existing score required adjustment. Averaging the two judges reduced absolute error relative to the audited individual judge in 48/82 groups recommended for adjustment (58.54%), although residual discrepancies remained. Across 162 unique response–dimension pairs, the averaged scores had an MAE of 0.399 points against the historical human mean. The current RC protocol thus offers a practical, auditable approximation to human judgments without introducing a separate, uncalibrated weight-estimation stage. Results also motivate that future work could develop and independently validate low-cost, high-fidelity weight estimation methods and context-sensitive aggregation rules.
Q3: sensitivity of different experts or judges
Response: We conducted additional evaluator-replacement experiments on the six models in the omni-modal track of Table 4. Specifically, we replaced Doubao with Grok as one of the RC judges and DeepSeek with Grok as one of the ERC experts, using grok-4.20-beta in both cases. The Spearman rank correlations between the original and replacement configurations are reported below.
| 指标 / Metric | Spearman ρ |
|---|---|
| MEC-lower | 0.943 |
| MEC-upper | 0.943 |
| EDD-intra | 0.886 |
| EDD-inter | 1.000 |
| RCD-intra | 0.886 |
| RCD-inter | 0.943 |
| CEC-lower | 1.000 |
| CEC-upper | 1.000 |
| ED-fusion | 1.000 |
| ED-speech | 1.000 |
| ED-face | 1.000 |
| ED-body | 0.943 |
| RC-Exp. | 0.943 |
| RC-Cha. | 1.000 |
| RC-Rel. | 0.943 |
Following these replacements, the average ERC ambiguity rate decreased by 1.82%, valid transition-pair coverage increased by 2.33%, and the RC drop rate decreased by 1.76%. These results suggest that model rankings remain relatively stable under the tested substitutions, with Spearman correlations ranging from 0.886 to 1.000 and modest changes in coverage.
For reproducibility, we nevertheless recommend using the evaluator models and versions specified in our evaluation protocol. Any substitutions or version changes should be explicitly documented to support transparent interpretation and comparison of results.
Q5: challenging errors in preliminary experiment
Response: We thank the reviewer for suggesting a more challenging evaluation! The preliminary experiment primarily aimed to assess whether existing evaluation protocols are sensitive to semantic-level errors. We therefore constructed 100 mismatched samples by randomly replacing the multimodal semantic descriptions.
Upon further inspection, 23 of these samples retained emotional overlap with their corresponding ground-truth (GT) descriptions, potentially indicating subtle emotion substitutions. Within this subset, the comparison results are as follows:
| Metrics | GT vs wrong(win/tie/lose/invalid) |
|---|---|
| CER | 5/14/4/0 |
| WavLM SIM | 14/0/9/0 |
| Character Consistency | 12/0/11/0 |
| Conversation Ability | 13/0/10/0 |
| Role-playing Attractiveness | 13/0/10/0 |
| Avg. of SF&SN | 8/3/12/0 |
| Avg. of PC&EA | 10/2/11/0 |
| Avg. of PeC&KC | 6/5/12/0 |
| SSIM | 10/0/13/0 |
| CPBD | 12/0/11/0 |
| Sync_cf | 4/0/10/9 |
| Avg. of Content Accuracy | 6/11/6/0 |
| Avg. of Style Accuracy | 7/7/9/0 |
| Avg. of MEC | 11/11/1/0 |
| Avg. of RC | 13/10/0/0 |
These results provide limited, preliminary evidence that MERRY may be able to better distinguish some semantic differences even when the descriptions partially share emotional content.
Q7: evaluation cost
Response: We reviewed the historical quota consumption and found that the total evaluation API cost (excluding generation) of the entire benchmark (including results in Table 4 and 5) is approximately 700-800 dollars. On average, each model requires an evaluation cost of 60-70 dollars after generating responses.
Q8: rare emotion performance
Response: We thank the reviewer for this constructive suggestion! We clarify that the validation results in Table 14 already include samples of frequent and rare emotions and quantify their consistency with human consensus labels respectively. The results do not show a strictly monotonic relationship between category frequency and recognition performance: several low-frequency emotions achieve F1 scores comparable to, or higher than, those of more frequent categories, while performance also varies within both frequency groups.
W1: validity of CEC & Q1: direct human agreement on CEC/ED
Thank you for these related comments! We agree that high cross-modal agreement does not necessarily imply accurate emotional expression. CEC and ED assess complementary properties of multimodal responses: cross-modal emotional consistency and emotional discrepancy (clarity), respectively.
As explained in the second paragraph of the Introduction (lines 53–78), modality-specific generators often process their corresponding semantic descriptions independently. Accordingly, our semantic-level evaluation requires each modality-specific description to independently support the same underlying emotions. This requirement motivates a distinction between two kinds of justified cross-modal contrast: those in which each modality independently supports the shared underlying emotions, and those whose plausibility depends on joint interpretation without such independent support.
To provide direct human validation on CEC and ED, we added an evaluation in which three annotators independently rated each response on five-point scales. Lower human ED scores indicate clearer emotional expression, whereas higher human CEC scores indicate greater consistency. They first provided modality-specific ED ratings, then directly assessed fusion-level ED and CEC, and annotated whether each response exhibited a justified cross-modal contrast. For responses exhibiting such contrasts, they were additionally instructed, when assigning CEC scores, to consider whether each modality-specific description independently supported the same underlying emotions inferred from the response as a whole.
For each metric, we averaged the three annotators’ ratings for each response and grouped responses into bins of width 0.5. Across bins, aggregate automatic ED and CEC showed strong positive Pearson and Spearman correlations with the corresponding mean human ratings. We further examined the 33 responses identified by at least two annotators as exhibiting justified cross-modal contrasts. Within this subset, responses with higher mean human CEC ratings (>3) still had obvious higher aggregate automatic CEC values than those with lower ratings (≤3), for both the lower and upper estimates. This result supports CEC's ability to distinguish justified contrasts under the independent-support criterion, at the group level. These findings provide direct evidence of alignment with human judgments of emotional clarity and cross-modal consistency at the aggregate level, consistent with the intended scope of ED and CEC.
The detailed protocol and results are reported in Appendix C, Tables 13 and 14.
W2: number of samples in human agreement exp
Response: We thank the reviewer for highlighting this concern! We agree that the limited sample size constrains the validation of evaluation reliability across diverse emotional contexts. Our choice of 100 samples per study was informed by related published studies using approximately 50–200 samples for human agreement evaluation. We used random sampling to reduce selection bias and sought to balance contextual coverage with a manageable annotation workload for each evaluator. Nevertheless, this design does not ensure adequate coverage of all emotional contexts, particularly rare or complex emotional expressions. We will explicitly acknowledge this limitation in the revised manuscript.
W4: generalizability of the benchmark
Response: We agree that current benchmark may have limited generalizability to broader multilingual and real-world interactive settings. Our experimental conclusions should therefore be interpreted within such evaluated setting. To broaden language coverage, we plan to provide an additional test set constructed from English-grounded multimodal emotional dialogue data as part of the dataset release. This extension will support further assessment of cross-lingual applicability.
Regarding the number of test roles, we separate training and test data at the TV-series level to reduce leakage from shared characters and narrative contexts. Although this split limits the number of test roles, the evaluation still reveals limitations in multimodal emotional and role consistency across the characters and contexts examined.
We will explicitly acknowledge this limitation in the revised manuscript.
W5: Missing references
Response: Thank you for the reminder! Both of these studies are valuable pieces of related work. We have added them and included citations in the main text (line 38 for E3RG, line 119 for Dynamic Persona Coherence).