acceptodds
Under review as a conference paper at ICLR 2027

EmoTrace-Bench: Diagnosing Emotional Expression in Text-to-Video Generation through Controlled Cues and Intensity

Abstract

Text-to-video (T2V) models are increasingly used to generate videos with targeted emotional expressions, yet existing evaluation methods mainly rely on holistic video–text alignment and provide limited insight into whether failures arise from target-emotion expression, intensity control, or specific affective cues. We introduce EmoTrace-Bench, a fine-grained diagnostic benchmark that controls emotion intensity and selected affective cues in prompts and evaluates generated videos at multiple levels. EmoTrace-Bench contains 448 controlled prompts covering eight emotions, two intensity levels, and diverse cue conditions. Using 12 T2V models, we generate 5,376 videos and evaluate target-emotion presence, perceived intensity, and ten audiovisual dimensions of emotional expression with an automated evaluator calibrated and validated against human annotations on a subset of the data. Results reveal substantial differences across models and prompt designs. Prompts that explicitly describe a character’s emotional reaction are followed more reliably than prompts that describe only events or scenes, whereas adding more affective cues or using more explicit cinematic audiovisual language does not consistently improve emotional generation. These results illustrate how aggregate alignment scores may overlook localized differences in target-emotion expression, intensity control, and cue realization. EmoTrace-Bench provides a traceable and interpretable framework for diagnosing emotional video generation without modifying existing generation pipelines, and we release the prompts, annotation guidelines, evaluation protocols, and automated scores.

Primary area:
datasets and benchmarks
Keywords:
Text-to-Video Generation, Video Generation Evaluation, Emotion Expression, Emotion Intensity, Affective Computing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HvRWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EmoTrace-Bench, a diagnostic benchmark for emotional expression in text-to-video generation. It contains 448 controlled prompts covering eight emotions, two intensity levels, and multiple affective cue conditions, and evaluates 5,376 videos generated by twelve models. Using automated evaluation validated against human annotations, the authors separately assess target-emotion presence, perceived intensity, and ten audiovisual cue dimensions. The results suggest that explicit character reactions are generally more effective than scene- or event-only descriptions, while adding affective cues does not consistently improve performance.

Critical strengths

The problem has practical significance. Emotional generation can fail through an incorrect emotion category, inadequate intensity control, or an inappropriate mode of expression. A single video–text similarity score cannot readily distinguish these failures. Evaluating these aspects separately is a well-motivated and valuable research direction.

Critical weaknesses

W1: The referent of the target emotion is insufficiently specified. The benchmark jointly evaluates character expressions and behaviors, scene atmosphere, and audiovisual form. However, emotion can refer to the affective state of a character, the affective tone of the environment, or the emotion we experience while watching the video. It remains unclear whether the shared target-emotion label conflates these objectives and whether this ambiguity affects model performance.

W2: The central motivation that aggregate metrics overlook localized failures lacks direct experimental validation. The paper argues that holistic video–text alignment metrics cannot distinguish failures in emotion category, intensity, and affective cue realization. However, the experiments primarily report results from the proposed evaluation framework without directly comparing it with representative aggregate metrics.

Questions for authors

See weaknesses.

Reviewer_R8PgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EmoTrace-Bench, a benchmark for evaluating emotional expression in text-to-video generation. It consists of 448 controlled prompts covering eight emotions, two intensity levels, and 14 prompt templates. The benchmark evaluates generated videos in terms of target-emotion accuracy, intensity control, and ten audiovisual expression dimensions.

The authors evaluate 12 T2V models on 5,376 generated videos using Gemini 3.1 Pro, with automated evaluation validated against human annotations on 1,120 videos. The experiments show that models struggle more with emotion intensity control than emotion expression, that explicitly describing character reactions substantially improves performance, and that adding more affective cues does not consistently lead to better results.

Critical strengths
  1. Careful experimental design. The benchmark uses controlled prompt variations to study emotion intensity and affective cues. The matched prompt pairs allow relatively systematic comparisons of how T2V models respond to different emotional instructions.

  2. Extensive evaluation and human validation. The authors evaluate 12 T2V models on 5,376 generated videos and collect human annotations for 1,120 videos to validate the automated evaluator. The reported agreement between human and automated judgments provides reasonable confidence in the evaluation results.

Critical weaknesses

Limited novel insights and insufficient analysis of interesting findings.

Despite the extensive experiments, most of the main findings are relatively intuitive, such as the benefits of explicitly describing emotional reactions and the difficulty of controlling emotion intensity. I learned little new from these observations.

More importantly, the paper provides limited analysis of the few potentially interesting findings. For example, the authors report a substantial performance gap between positive and negative emotions, with anger showing particularly poor performance, but do not investigate why this occurs. Similarly, the observation that adding more affective cues does not consistently improve performance receives little further explanation.

Given that the paper already offers relatively few surprising findings, I believe a deeper analysis of these results is particularly important. The proposed fine-grained diagnostic framework could potentially help explain these behaviors, but the paper largely stops at reporting performance differences rather than identifying their underlying causes. This significantly limits the scientific insights and practical value of the benchmark.

Questions for authors
  1. Diagnostic value of the proposed framework. What additional insights does the proposed 10D evaluation provide beyond overall emotion and intensity accuracy? For example, can the authors use these dimensions to explain why anger performs substantially worse than other emotions, or why adding more affective cues does not consistently improve performance?

  2. Evaluation methodology. What motivates using Gemini 3.1 Pro as the primary evaluator across all ten dimensions rather than incorporating specialized evaluation models? Additionally, could the authors clarify the implementation of the visual-dynamics and audio auxiliary modules and whether these modules improve agreement with human annotations compared to using Gemini alone?

Reviewer_9fApOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EmoTrace-Bench splits emotional text-to-video generation into three levels: whether the target emotion is present, whether perceived intensity is weak or strong, and which of ten audiovisual channels supply evidence. The 448 controlled prompts cover 8 emotions, 2 intensity levels, and 14 templates. Twelve models generate 5,376 videos. Humans annotate 1,120 videos from three models to calibrate and test the automated evaluator.

Critical strengths
  1. The minimal-difference prompt design is why changes can be attributed to a specific edit. Strong/weak pairs change only preregistered intensity wording, and 224/224 pairs pass character-level checks. “Writing a character reaction adds about 45 points” can be attributed to that prompt change rather than to a simultaneous change of scene or emotion.

  2. The automatic evaluator is not treated as ground truth. Development-set Fleiss’ κ is 0.65 for emotion, 0.50 for intensity, and 0.60 for the ten channels. Joint three-way agreement on the held-out test set is 87.1% (κ=0.81) against the reviewed reference. A second backbone agrees with the main evaluator at κ=0.62–0.68 on 224 videos. I trust the qualitative gaps, not the small differences among the 12 models.

Critical weaknesses
  1. The 12-model main table is produced almost entirely by one automated evaluator, and human labels cover only three generators. Human κ for intensity is only about 0.50, and intensity control is the central negative result. The full 5,376 videos do not receive the same human audit. This is what keeps me at weak acceptance rather than clear acceptance.

  2. The prompt distribution is narrow: short, single-scene English prompts. “Adding every cue does not help” holds inside this template family. The abstract still reads like a general fact about model ability.

Questions for authors
  1. Can you relabel a stratified subset of all 12 models with the same human protocol and report whether the intensity-control ranking changes?

  2. Do Seedance-2.5 and Veo-3.1 remain the top group on a human subset?

  3. Do you try to use Hume AI which provides an accurate and more fine-grained emotion detection tool for the evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.