acceptodds
Under review as a conference paper at ICLR 2027

MERRY: Semantic-Level Evaluation of Multimodal Emotional and Role Consistencies for Role-Playing Agents

Abstract

Multimodal Role-Playing Agents (MRPAs) are attracting increasing attention due to their ability to deliver more immersive multimodal emotional interactions. However, existing studies typically evaluate response content with textual role-playing benchmarks, while assessing multimodal expressions mainly through modality-synthesis metrics. This evaluation paradigm leaves a critical diagnostic gap: the semantics of multimodal expressions are only assessed after modality generation, making it difficult to determine whether an error originates from multimodal semantic response planning or from final modality realization. To this end, we propose MERRY, a semantic-level evaluation framework for assessing **M**ultimodal **E**motional and **R**ole consistencies of **R**ole-pla**y**ing agents. This framework extends existing evaluation dimensions to multimodal semantic responses, and introduces multimodal emotional interpretability to assess whether generated semantic descriptions support a coherent emotional interpretation. It also introduces a bidirectional evidence-finding scoring protocol for Large-Language-Model-as-a-Judge scoring, improving the alignment with human evaluators. To support evaluation, we further construct MERRY, which contains 25,507 samples with a TV-series-level separated test split. Extensive evaluations with MERRY show that current MRPAs still have substantial room for improvement in multimodal emotional and role consistency on semantic level.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Role-Playing Agents, Dataset and Benchmark, Semantic-Level Evaluation, Emotional Consistency, Role Consistency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Yoc7OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MERRY, a semantic-level evaluation framework for assessing emotional and role consistency in multimodal role-playing agents. It introduces metrics for cross-modal emotional consistency and emotional discrepancy, along with a bidirectional evidence-finding LLM-as-a-Judge protocol. The paper further construct MERRY-Data, and evaluate multiple multimodal models to identify limitations in emotional expression and role consistency. Experiments under different datasets show gains over baselines.

Critical strengths

1.The paper addresses an underexplored problem in multimodal role-playing evaluation by focusing on semantic consistency before modality synthesis. 2.MERRY introduces several metrics and an evidence-based evaluation protocol, providing a more fine-grained perspective on emotional and role consistency. 3.The newly constructed MERRY-Data contains diverse multimodal semantic annotations, and experiments across multiple models provide useful insights into current limitations.

Critical weaknesses

1.The validity of the proposed emotional consistency metrics needs further justification. High cross-modal agreement does not necessarily indicate accurate emotional expression, especially when different modalities intentionally convey contrasting emotions. 2.The evaluation relies heavily on LLM-based judgments. Although human agreement studies are provided, they involve only 100 samples per study, which may be insufficient to validate the reliability of the proposed metrics across diverse emotional contexts. 3.The proposed metrics may favor overly consistent or simplified emotional descriptions. For example, MERRY fine-tuning increases CEC substantially from 0.586 to 0.934, while MEC improves only slightly from 0.264 to 0.291, raising concerns about the relationship between cross-modal consistency and emotional correctness. 4.The generalizability of the benchmark is limited. MERRY-Data is derived from Chinese TV-series dialogues with a relatively small number of test roles, leaving its applicability to broader multilingual and real-world interactive scenarios unclear. 5.Missing references about consistency remaining emotional and role playingb system: [1]Ronghao Lin, Shuai Shen, Weipeng Hu, Qiaolin He, Aolin Xiong, Li Huang, Haifeng Hu, and Yap-peng Tan. 2025. E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Model. In Proceedings of the 33rd ACM International Conference on Multimedia (MM '25). Association for Computing Machinery, New York, NY, USA, 14006–14013. [2]Yirui QI, Xiaoming Zhang, Ruilin Zeng, Mengyao Liu, Ziyi Zhou, Dezhuang Miao, Bingyu Yan, and Zhenyu Guan. 2026. Beyond Static Persona Consistency: Dynamic Persona Coherence in LLM Role-Playing. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28942–28956, San Diego, California, United States. Association for Computational Linguistics.

Questions for authors

1.Could the authors report the correlation between the proposed CEC/ED metrics and direct human judgments of overall multimodal emotional coherence? 2.What is the impact of excluding the fused emotion channel when computing CEC, given that this channel already incorporates information from the other modalities? 3.How sensitive are the evaluation results to different combinations of ERC experts or LLM judges? 4.Can the authors provide an analysis of whether the proposed metrics are affected by the length or level of detail of generated semantic descriptions? 5.Could the authors evaluate more challenging semantic errors, such as subtle emotion substitutions, rather than only randomly mismatched semantic descriptions? 6.Can the authors provide more examples of cases where the proposed metrics disagree with human judgments and discuss the reasons? 7.What is the computational cost of evaluating the complete benchmark, considering the multiple LLM-based emotion recognition and judgment calls? 8.Could the authors further analyze performance differences between frequent and rare emotion categories to assess whether the benchmark captures fine-grained emotional understanding?

6 comments

W1: validity of CEC & Q1: direct human agreement on CEC/ED

Thank you for these related comments! We agree that high cross-modal agreement does not necessarily imply accurate emotional expression. CEC and ED assess complementary properties of multimodal responses: cross-modal emotional consistency and emotional discrepancy (clarity), respectively.

As explained in the second paragraph of the Introduction (lines 53–78), modality-specific generators often process their corresponding semantic descriptions independently. Accordingly, our semantic-level evaluation requires each modality-specific description to independently support the same underlying emotions. This requirement motivates a distinction between two kinds of justified cross-modal contrast: those in which each modality independently supports the shared underlying emotions, and those whose plausibility depends on joint interpretation without such independent support.

To provide direct human validation on CEC and ED, we added an evaluation in which three annotators independently rated each response on five-point scales. Lower human ED scores indicate clearer emotional expression, whereas higher human CEC scores indicate greater consistency. They first provided modality-specific ED ratings, then directly assessed fusion-level ED and CEC, and annotated whether each response exhibited a justified cross-modal contrast. For responses exhibiting such contrasts, they were additionally instructed, when assigning CEC scores, to consider whether each modality-specific description independently supported the same underlying emotions inferred from the response as a whole.

For each metric, we averaged the three annotators’ ratings for each response and grouped responses into bins of width 0.5. Across bins, aggregate automatic ED and CEC showed strong positive Pearson and Spearman correlations with the corresponding mean human ratings. We further examined the 33 responses identified by at least two annotators as exhibiting justified cross-modal contrasts. Within this subset, responses with higher mean human CEC ratings (>3>3) still had obvious higher aggregate automatic CEC values than those with lower ratings (≤3\leq3), for both the lower and upper estimates. This result supports CEC's ability to distinguish justified contrasts under the independent-support criterion, at the group level. These findings provide direct evidence of alignment with human judgments of emotional clarity and cross-modal consistency at the aggregate level, consistent with the intended scope of ED and CEC.

The detailed protocol and results are reported in Appendix C, Tables 13 and 14.

W2: number of samples in human agreement exp

Response: We thank the reviewer for highlighting this concern! We agree that the limited sample size constrains the validation of evaluation reliability across diverse emotional contexts. Our choice of 100 samples per study was informed by related published studies using approximately 50–200 samples for human agreement evaluation. We used random sampling to reduce selection bias and sought to balance contextual coverage with a manageable annotation workload for each evaluator. Nevertheless, this design does not ensure adequate coverage of all emotional contexts, particularly rare or complex emotional expressions. We will explicitly acknowledge this limitation in the revised manuscript.

W4: generalizability of the benchmark

Response: We agree that current benchmark may have limited generalizability to broader multilingual and real-world interactive settings. Our experimental conclusions should therefore be interpreted within such evaluated setting. To broaden language coverage, we plan to provide an additional test set constructed from English-grounded multimodal emotional dialogue data as part of the dataset release. This extension will support further assessment of cross-lingual applicability.

Regarding the number of test roles, we separate training and test data at the TV-series level to reduce leakage from shared characters and narrative contexts. Although this split limits the number of test roles, the evaluation still reveals limitations in multimodal emotional and role consistency across the characters and contexts examined.

We will explicitly acknowledge this limitation in the revised manuscript.

W5: Missing references

Response: Thank you for the reminder! Both of these studies are valuable pieces of related work. We have added them and included citations in the main text (line 38 for E3RG, line 119 for Dynamic Persona Coherence).

W3: CEC may favor simplified emotional descriptions? & Q4: how metrics are affected by length

Response: Thank you for raising this question! With regard to the importance and validity of CEC , we have already responded in sections W1 and Q1.

Regarding the effects of fine-tuning, we observed that improvements across different metrics are not uniform following straightforward fine-tuning: given the same data volume and computational budget, the gains for MEC are notably smaller than those for CEC. This may be due to a disparity in the difficulty levels of the two metrics. Consequently, as noted in the final sentence of our analysis in Section 5.2, more targeted training approaches—such as Reinforcement Learning (RL)—may be necessary and useful in the future.

To examine length sensitivity, we additionally analyzed all nine zero-shot models in Table 4. For each model separately, we divided responses into 20 bins based on the total non-whitespace character count. Within each bin, we recomputed CEC and ED, and then calculated Pearson’s rr between each metric and the actual mean response length across the 20 bins:

ModelCEC lowerCEC upperED fusionED speechED facialED body
OmniCharacter-7B0.5750.673−0.0110.840−0.0860.166
Doubao-1.5-Vision0.7640.5640.5370.4380.3600.429
GPT-5-Chat0.5770.4840.5720.0350.3150.383
Gemini-2.5-Pro0.6400.5230.4750.3490.4340.184
Qwen3-Omni-30B-A3B0.9190.914−0.680−0.788−0.807−0.805
MiniCPM-o-4.50.8970.892−0.825−0.779−0.790−0.689
MiniCPM-o-2.60.9170.907−0.655−0.813−0.791−0.565
Qwen2.5-Omni-7B0.9000.887−0.461−0.314−0.319−0.326
Phi-4-Multimodal-5.6B0.8960.856−0.729−0.235−0.267−0.113

Results shows that CEC is positively associated with length for evaluated model, while ED shows model-dependent directions. This indicates that high CEC and lower ED are not solely limited to simple, brief descriptions. Notably, these correlations do not simply imply that CEC has a stronger preference for longer responses, as the specific CEC values still vary across different models.

Q2: CEC without fusion

Response: We include the fusion channel so that CEC captures both agreement among independently inferred modality-specific emotions and their alignment with the emotional interpretation of the complete response. The fusion channel serves as a holistic reference derived from all semantic fields, including the textual content. This fused interpretation is also used in computing other emotional evaluation metrics.

Computing CEC from only the three modality-specific channels measures their mutual agreement but does not directly assess alignment with the complete response. It can therefore overlook cases where the three channels agree with one another but conflict with the joint interpretation, without a coherent explanation for that discrepancy.

Our supplementary human-agreement study provides empirical support for this design choice. As shown in Table 13, excluding the fusion channel reduces agreement between CEC and direct human judgments. This finding supports retaining the holistic reference when assessing cross-modal emotional consistency.

Q6: why disagree with human judgments

Response: Thank you for this suggestion! We have added a targeted human audit of RC evidence and score aggregation in Appendix G. Critical evidence was the most frequently endorsed reason for adjusting individual-judge scores (61/82 groups, 74.39%). These cases highlight potential limitations of evidence counting in capturing the relative importance of supporting and conflicting evidence. But results also found that the presence of critical evidence alone did not determine whether an existing score required adjustment. Averaging the two judges reduced absolute error relative to the audited individual judge in 48/82 groups recommended for adjustment (58.54%), although residual discrepancies remained. Across 162 unique response–dimension pairs, the averaged scores had an MAE of 0.399 points against the historical human mean. The current RC protocol thus offers a practical, auditable approximation to human judgments without introducing a separate, uncalibrated weight-estimation stage. Results also motivate that future work could develop and independently validate low-cost, high-fidelity weight estimation methods and context-sensitive aggregation rules.

Q3: sensitivity of different experts or judges

Response: We conducted additional evaluator-replacement experiments on the six models in the omni-modal track of Table 4. Specifically, we replaced Doubao with Grok as one of the RC judges and DeepSeek with Grok as one of the ERC experts, using grok-4.20-beta in both cases. The Spearman rank correlations between the original and replacement configurations are reported below.

指标 / MetricSpearman ρ
MEC-lower0.943
MEC-upper0.943
EDD-intra0.886
EDD-inter1.000
RCD-intra0.886
RCD-inter0.943
CEC-lower1.000
CEC-upper1.000
ED-fusion1.000
ED-speech1.000
ED-face1.000
ED-body0.943
RC-Exp.0.943
RC-Cha.1.000
RC-Rel.0.943

Following these replacements, the average ERC ambiguity rate decreased by 1.82%, valid transition-pair coverage increased by 2.33%, and the RC drop rate decreased by 1.76%. These results suggest that model rankings remain relatively stable under the tested substitutions, with Spearman correlations ranging from 0.886 to 1.000 and modest changes in coverage.

For reproducibility, we nevertheless recommend using the evaluator models and versions specified in our evaluation protocol. Any substitutions or version changes should be explicitly documented to support transparent interpretation and comparison of results.

Q5: challenging errors in preliminary experiment

Response: We thank the reviewer for suggesting a more challenging evaluation! The preliminary experiment primarily aimed to assess whether existing evaluation protocols are sensitive to semantic-level errors. We therefore constructed 100 mismatched samples by randomly replacing the multimodal semantic descriptions.

Upon further inspection, 23 of these samples retained emotional overlap with their corresponding ground-truth (GT) descriptions, potentially indicating subtle emotion substitutions. Within this subset, the comparison results are as follows:

MetricsGT vs wrong(win/tie/lose/invalid)
CER5/14/4/0
WavLM SIM14/0/9/0
Character Consistency12/0/11/0
Conversation Ability13/0/10/0
Role-playing Attractiveness13/0/10/0
Avg. of SF&SN8/3/12/0
Avg. of PC&EA10/2/11/0
Avg. of PeC&KC6/5/12/0
SSIM10/0/13/0
CPBD12/0/11/0
Sync_cf4/0/10/9
Avg. of Content Accuracy6/11/6/0
Avg. of Style Accuracy7/7/9/0
Avg. of MEC11/11/1/0
Avg. of RC13/10/0/0

These results provide limited, preliminary evidence that MERRY may be able to better distinguish some semantic differences even when the descriptions partially share emotional content.

Q7: evaluation cost

Response: We reviewed the historical quota consumption and found that the total evaluation API cost (excluding generation) of the entire benchmark (including results in Table 4 and 5) is approximately 700-800 dollars. On average, each model requires an evaluation cost of 60-70 dollars after generating responses.

Q8: rare emotion performance

Response: We thank the reviewer for this constructive suggestion! We clarify that the validation results in Table 14 already include samples of frequent and rare emotions and quantify their consistency with human consensus labels respectively. The results do not show a strictly monotonic relationship between category frequency and recognition performance: several low-frequency emotions achieve F1 scores comparable to, or higher than, those of more frequent categories, while performance also varies within both frequency groups.

Reviewer_6dxAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MERRY, a semantic-level evaluation framework for the emotional and role consistencies of Multimodal Role-Playing Agents before modality synthesis. It extends existing emotional consistency metrics to multimodal semantic responses, introduces Cross-modal Emotional Consistency (CEC) and Emotional Discrepancy (ED), and uses a bidirectional evidence-finding protocol for Role Consistency (RC). The authors construct MERRY-Data from CPED, containing 25,507 samples with a TV-series-level separated test split. Evaluation includes nine models under the zero-shot setting, fine-tuning experiments, and three human agreement studies.

Critical strengths

1.MERRY offers a useful and original perspective on evaluating multimodal emotional and role consistencies before modality synthesis. The experimental results support the effectiveness of this evaluation framework.

2.The framework assesses complementary aspects of emotional and role consistency. The experiments include multiple models, fine-tuning comparisons, and human validation, making the study fairly complete.

3.The paper is well written and clearly organized, with clear explanations of the task and evaluation methodology that make the proposed framework easy to follow.

Critical weaknesses

1.CEC measures cross-modal emotional agreement, but a low score may reflect either an error in the model output or a justified cross-modal contrast in the ground-truth response. The metric alone does not distinguish these cases.

2.RCD compares average distances between roles, but this aggregation may mask changes in individual pairwise distances. All evaluated models obtain scores close to zero, making it unclear how well RCD distinguishes their ability to preserve emotional differences between roles.

3.RC uses GPT and Doubao judges, and the evaluated models include variants from both families. Potential preferences for responses from the same model family are not examined, raising a concern about the fairness of these comparisons.

Questions for authors

Please address the concerns listed in my weaknesses.

7 comments

W1: CEC issue

Response: Thank you for raising this distinction! We clarify that CEC for a model-generated response is computed solely from the emotion labels inferred from that response; neither the ground-truth response nor its emotion labels enter the calculation. The emotion recognition procedure and CEC calculation are described on pp. 5 and 6, respectively.

Our design distinguishes between two kinds of justified cross-modal contrast: those in which each modality-specific description independently supports the same underlying emotions, and those that are justified when interpreted jointly but do not provide this independent support across modalities. This distinction matters because modality-specific generators often process their corresponding semantic descriptions independently (also been mentioned in Introduction). Accordingly, our semantic-level evaluation requires the intended underlying emotions to be supported by each modality’s independent description. A contrast may therefore be narratively justified while still receiving a low CEC score under this criterion. Conversely, differences in surface expression can be compatible with high CEC when each description independently supports the shared underlying emotions.

The new human evaluation supplemented in Appendix C directly examines this criterion. Three annotators independently rated CEC on five-point scales, and annotate whether each response exhibited a justified cross-modal contrast. For responses exhibiting such contrasts, they were additionally instructed, when assigning CEC scores, to consider whether each modality-specific description independently supported the same underlying emotions inferred from the response as a whole.

Across bins of width 0.5 formed from mean human ratings, automatic CEC correlated positively with human judgments, with Pearson’s r=0.857r=0.857 and 0.9050.905 for the lower and upper estimates, respectively (Table 13). Among the 33 responses identified by at least two annotators as exhibiting justified contrasts, the higher-rated group (>3>3, N=19N=19) had higher aggregate automatic CEC than the lower-rated group (≤3\leq3, N=14N=14): 0.5740.574 vs. 0.2260.226 for CEClowerCEC_{lower} and 0.6920.692 vs. 0.4140.414 for CECupperCEC_{upper} (Table 14). These results support the intended distinction: even among justified contrasts, auto CEC differentiates groups with higher versus lower human-rated CEC under the requirement of independent support for shared underlying emotions.

W2: RCD issue

Response: Thank you for highlighting this limitation! We agree that positive and negative deviations in individual role-pair distances can cancel in RCD. RCD inherits the signed aggregation principle used in EmoCharacter, assessing whether emotional differentiation across roles is expanded or contracted on average relative to ground truth.

Motivated by your concern, we additionally report pairwise mean absolute error (PairMAE), which measures the magnitude of errors in corresponding role-pair distances, irrespective of direction and without cancellation between positive and negative deviations. RCD and PairMAE thus capture complementary properties: the direction of net change in average role differentiation and the magnitude of individual role-pair distance errors.

We report 95% role-pair bootstrap intervals for RCD alongside PairMAE point estimates in Appendix H. The results in Table 21 show that, despite the small numerical values of RCD, several intervals lie entirely above or below zero, revealing model-specific patterns of expansion and contraction in average emotional differentiation across roles. PairMAE additionally reveals pairwise discrepancies that are obscured by near-zero RCD values. This additional analysis clarifies the interpretation and limitations of RCD while providing a complementary assessment of how accurately individual role-pair distances are preserved.

W3: potential RC judge preferences

Response: Thank you for raising this concern! To address it, we focused on the vision track, which contains the two evaluated generators whose families overlap with those of the RC judges. To assess the sensitivity of vision-track RC comparisons to judge choice, we compared GPT-5 and Doubao-1.5v using three scoring configurations: Doubao-1.6 alone, GPT-5.2 alone, and the mean of both judges’ scores.

For each dimension, all three judge configurations use the same subset of samples with valid scores from both judges for both generators. All scores and differences were recomputed on these shared subsets. Valid rate denotes the proportion of these jointly valid samples in the full test set (N = 1,690). Entries report mean score differences (GPT-5 minus Doubao-1.5v) with pointwise 95% confidence intervals, which were estimated using 10,000 paired bootstrap resamples of dialogue groups within each TV series, keeping samples sharing original or merged dialogue IDs together.

DimensionValid rateDoubao-1.6 judgeGPT-5.2 judgeMean of both judges
Exp.48.64%+0.013 [−0.030, +0.058]−0.078 [−0.134, −0.023]−0.032 [−0.075, +0.008]
Cha.99.05%+0.061 [+0.024, +0.099]−0.010 [−0.051, +0.031]+0.025 [−0.006, +0.058]
Rel.85.74%+0.012 [−0.015, +0.041]+0.008 [−0.036, +0.051]+0.010 [−0.018, +0.038]

When both judges are averaged, the estimated differences are close to zero, and the 95% confidence intervals include zero for all three dimensions. The directions of the averaged differences above also remain consistent with Table 4 (Doubao-1.5v scores slightly higher on Exp., whereas GPT-5 scores slightly higher on Cha. and Rel.), even after restricting the comparison to jointly valid samples. Overall, the combined evaluation shows no clear advantage for either generator on the jointly valid samples.

I appreciate the authors’ detailed responses and additional experiments, which have addressed some of my concerns, particularly regarding the validity and interpretation of CEC and RCD. However, I still have reservations about the robustness of the RC evaluation. While the cross-judge analysis helps alleviate concerns about model-family preferences, the results reveal notable differences between individual judges, especially in the Experience and Characteristics dimensions. Specifically, the GPT judge significantly favors Doubao-1.5v in Experience, whereas the Doubao judge significantly favors GPT-5 in Characteristics. Although the averaged scores show no clear advantage for either model, these discrepancies suggest that the relative comparisons may be sensitive to judge selection. I encourage the authors to further investigate this issue by reporting confidence intervals for the differences in relative model assessments between judges. If this remaining concern can be adequately addressed, I would be inclined to evaluate the paper more favorably.

Reviewer_ZANyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MERRY evaluates emotional and role semantics before audiovisual synthesis, introducing series-separated MERRY-Data, diagnostic emotion metrics, and bidirectional evidence-based RC scoring. It evaluates multiple models, adaptation, and human agreement.

Critical strengths
  • Separating semantic planning from modality realization offers a clear diagnostic scope.
  • Input tracks, ERC visibility, justified emotional contrast, and conditional results are explicitly specified.
Critical weaknesses
  • The new metrics need more direct validation against human judgments. Tables 7 and 12 mainly test agreement on emotion labels. They do not directly test whether ED identifies unclear expression or CEC identifies inconsistencies across modalities. Human judgments targeting these two questions would strengthen the evaluation.
  • Role scoring may depend too much on evidence counts. Under Table 3's rules, adding several ordinary supporting details can raise the score while an important contradiction remains. Existing human agreement is useful, but targeted examples are needed to test this behavior.
  • Some statistical results do not directly support their interpretation. Appendix A uses the proportion of ratings above 3 to discuss human preference for correct examples. That proportion does not establish pairwise preference. Please report direct correct-versus-incorrect comparisons, including ties and uncertainty.
  • Another important thing, character profiles need further factual checks. Figure 5 describes Xia Yu as Liu Xing's maternal half-brother, although they are stepbrothers. Since profiles also guide scoring, errors could affect both model outputs and evaluation. Please check whether this error occurs in the actual data and report a profile audit.
Questions for authors

Please refer to the weaknesses above. I currently rate my score as weak rejection, but would consider raising it.

4 comments

W1: direct human agreement on ED/CEC

Response: Thank you for highlighting the need for direct human validation! We added a human evaluation in which three annotators independently assessed each response on five-point scales targeting the clarity of emotional expression and cross-modal emotional consistency. Lower human ED scores indicate clearer emotional expression, whereas higher human CEC scores indicate greater consistency.

We averaged the three annotators' ratings for each response and grouped responses into bins of width 0.5 based on these mean ratings. Across bins, the aggregate automatic ED and CEC metrics showed strong positive Pearson and Spearman correlations with the corresponding mean human ratings (Table 13).

We also examined the 33 responses identified by at least two annotators as exhibiting justified cross-modal contrasts. As shown in Table 14, within this subset, responses with mean human CEC ratings above 3 had obvious higher aggregate automatic CEC values than those with ratings at or below 3, for both the lower and upper CEC estimates. These findings provide additional evidence of alignment with human judgments at the aggregate level.

The detailed protocol and results are appended to Appendix C.

W2: evidence counts and important contradictions.

Response: Thank you for identifying this specific failure mode! We agree that the overall human-agreement results in Table 7(b) should be complemented by a targeted examination of whether important contradictions are adequately reflected in RC scores. We have therefore added an evidence audit conducted by independent annotators, with the full protocol and results reported in Appendix G.

The audit found that

  1. Most supporting and conflicting evidence items were judged valid and substantively influential in the judgment for the target RC dimension, helping address the concern about accumulating trivial supporting details.

  2. Critical evidence was the most frequently endorsed reason for adjusting scores from individual judges, but its presence alone did not determine whether an existing score required adjustment.

  3. Averaging also reduced absolute error relative to the audited individual-judge score in a majority (58.54%) of groups recommended for adjustment.

  4. The targeted analysis also identified cases consistent with the reviewer’s concern. Among 29 groups originally scored 4 that contained valid critical conflicting evidence but no critical supporting evidence, annotators recommended decreasing 13 scores. For these 13 groups, judge averaging reduced absolute error relative to the historical human mean in 6, left it unchanged in 4, and increased it in 3. Averaging thus provided partial mitigation for individual judge, with interpretable discrepancies (explained in line 1595-1597 in Appendix G). Nevertheless, after deduplicating the groups into response–dimension pairs, most final averaged scores were within half a point of the historical human mean. The same held across the full audited sample.

  5. Even matching both evidence counts and mean importance ratings did not uniquely determine the adjustment direction. Whether scores need to be adjusted still partially depends on the specific case. It thereby fails to directly determine a universal reweighting rule, and motivates independent calibration of any proposed weighting rule in the future.

Taken together with the higher human agreement than explain-then-score reported in Table 7(b), these findings support retaining equal-weight aggregation of explicit evidence combined with judge averaging as the default RC protocol. It offers a practical, auditable approximation to the human reference without introducing a separate, uncalibrated weight-estimation stage. The results and analysis also make the reasons for remaining deviations explicit. Developing and independently validating low-cost but high-fidelity weight estimation methods and context-sensitive weighting rules remains a useful direction for further work.

W3: correct-versus-incorrect comparisons of Table 8 and 9

Response: Thank you for pointing this out! The metric previously reported as P(GT>Wrong)P(GT > Wrong) was, in fact, calculated on a combination-wise basis (e.g 100×100100 \times 100) rather than a pair-wise basis, which may have obscured the clarity of the comparison. Consequently, we have updated Tables 8 and 9 accordingly to focus on "correct-versus-incorrect" comparisons. The results still shows a clear difference between MERRY and other benchmarks.

Regarding the metric P(x>3)P(x > 3) in the original Table 9—which represents the probability of a score exceeding 3—our intention was to demonstrate that the generated videos exhibit sufficient fidelity and adherence to the specified semantic descriptions (whether derived from the ground-truth or "wrong" prompts), thereby confirming the semantic validity of the generated output. While faithfully maintaining the multimodal character style remains challenging for the current synthesis pipeline.

W4: figure 5's issue

Response: Thank you for pointing this out! We have re-checked the actual dataset and confirmed that this error does not exist. All profiles had already been carefully verified by 20 undergraduate students during the dataset construction process. As our primary focus had been on drafting the main text, we inadvertently overlooked the need to update certain legacy content in the appendix, which resulted in Figure 5 displaying the original, uncorrected profiles. We sincerely apologize for this oversight and have now updated Figure 5 in the manuscript to the latest version.

Thanks for the extra work on the human evaluation and evidence audit. The new ED/CEC study addresses my concern more directly, and the revised correct-versus-incorrect comparisons and profile checks are helpful. I also appreciate the detailed RC audit. It confirms that important contradictions can sometimes be under-penalized and that judge averaging only partly helps, so I still consider this a limitation. The audit does, however, make the behavior and trade-offs much clearer. Taking these improvements into account, I have raised my score to 3.

Thank you for your thoughtful reassessment and constructive feedback! We appreciate your recognition of the additional experiments and analyses and agree that, although judge averaging provides some mitigation, under-penalization of important contradictions remains a limitation. Your comments have helped us clarify the strengths and limitations of our evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.