- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EmoGenie, a benchmark of 6,400 AI-generated images for evaluating expressed and evoked emotions through recognition, comparison, description, and continuous assessment. The authors also propose EmoProbe, which combines emotion instruction tuning with regression heads for continuous affective prediction.
Critical strengths
-
The paper distinguishes between expressed and evoked emotions in AI-generated images, which is a meaningful direction for evaluating affective image generation and understanding.
-
EmoGenie provides a comprehensive benchmark covering emotion recognition, image-pair comparison, open-ended description, and continuous affective assessment. The combination of these tasks allows for a more thorough evaluation of MLLM emotion understanding.
-
The experiments cover 8 T2I models and 14 MLLM baselines, along with ablation studies across different backbones. The results reveal interesting differences in emotion generation and understanding across emotion categories and affective perspectives.
Critical weaknesses
-
The methodological novelty of EmoProbe is relatively limited. Its two-stage training mainly combines standard instruction tuning with dimension-specific regression heads, without introducing a substantially new approach to emotion understanding.
-
Although the benchmark contains 6,400 images, they are generated from only 800 prompts. The paper uses an image-disjoint train/test split but does not clearly establish whether prompts are also separated, raising concerns about potential semantic overlap between training and test data.
-
Evoked emotions are inherently subjective and may vary considerably across individuals. However, the benchmark relies primarily on majority voting and averaged human ratings, which may overlook individual differences and uncertainty in emotional responses.
-
EmoProbe is specifically trained on EmoGenie, while most baseline MLLMs are evaluated without equivalent task-specific training. This makes it difficult to determine whether the improvements come from the proposed method or simply from training on the benchmark data.
-
The reliability of human annotations needs further investigation, particularly for continuous dimensions such as arousal and dominance. Reporting human agreement or a noise ceiling would help determine how closely models can reasonably be expected to match human judgments.
-
The open-ended description task relies on another LLM (DeepSeek-v4-Flash) for evaluation, which may introduce evaluator bias. Additional human evaluation would help validate the reliability of these results.
Questions for authors
-
The benchmark contains 6,400 images generated from 800 prompts. Does the image-disjoint train/test split also ensure that images generated from the same prompt do not appear in both sets? If not, how might this semantic overlap affect the evaluation results?
-
Evoked emotions can vary substantially across individuals. Could the authors provide more detailed analysis of human annotation agreement, particularly for emotion categories and continuous VAD ratings? How reliable are the aggregated labels as ground truth?
-
EmoProbe is trained on EmoGenie, while most baseline MLLMs are evaluated without equivalent task-specific training. Could the authors clarify how much of the performance improvement comes from the proposed architecture versus training on the benchmark data?
-
The open-ended description task uses DeepSeek-v4-Flash as an evaluator. How reliable are these automated scores compared with human judgments, and could evaluator bias affect the model rankings?
3 comments
Q2 & W3 & W5. Human Annotation Agreement and Aggregated Targets.
Human annotation agreement. We examine agreement for emotion categories and continuous VAD ratings separately. Tab. 7 in App. B.2 reports the coefficients before QA construction. We use Gwet’s AC1 for nominal emotion categories and quadratic-weighted AC2 for the ordered VAD ratings. AC1 is appropriate for categorical judgments without an inherent ordering, whereas AC2 accounts for the magnitude of disagreements between rating levels.
| Affective perspective | Annotation attribute | Agreement measure | Agreement coefficient |
|---|---|---|---|
| Expressed emotion | Emotion category | AC1 | 0.682 |
| Evoked emotion | Emotion category | AC1 | 0.449 |
| Evoked emotion | Valence | Quadratic-weighted AC2 | 0.784 |
| Evoked emotion | Arousal | Quadratic-weighted AC2 | 0.798 |
| Evoked emotion | Dominance | Quadratic-weighted AC2 | 0.763 |
Reliability of aggregated category labels. During QA construction, an emotion-category label is retained only if it receives at least 60% of the votes and has more votes than every other category. We have further quantified the support and stability of these retained labels. Mean support is the average proportion of annotators selecting the retained label for each image. To assess stability, we removed each vote in turn and checked whether the original label remained the sole most-voted category. A tie or a different winning category was counted as a change.
| Affective perspective | Mean support for the retained label | Label preserved after removing one vote |
|---|---|---|
| Expressed emotion | 87.61% | 94.34% |
| Evoked emotion | 78.85% | 91.42% |
The stability calculation checks whether the winning label changes, rather than reapplying the 60% inclusion threshold after vote removal. These results indicate that the retained labels have substantial annotator support and are generally stable to the removal of an individual judgment.
Continuous VAD references. VAD reference scores are obtained by averaging human ratings for each image and dimension. During QA construction, targets with an inter-annotator SD above 2 are excluded to reduce uncertainty arising from substantial rating disagreements. EmoGenie-Assess inherits the retained targets and their corresponding mean scores. Together, the weighted agreement analysis and dimension-specific consistency screening support the use of the retained mean VAD ratings as continuous reference scores for evaluation.
Q3 & W4. Contribution of Task-Specific Training and Model Design.
Contribution of task-specific training. The baseline MLLMs in Tab. 4 are evaluated without additional training on EmoGenie. To examine the contribution of training on the benchmark data, we have further fine-tuned EEmo-Logic on the same EmoGenie-QA training set. Results for perception, description and continuous assessment are reported below:
| Model | Overall QA accuracy (%) ↑ | Description score (0–100) ↑ | Assess mean SRCC ↑ | Assess mean PLCC ↑ |
|---|---|---|---|---|
| EEmo-Logic, originally reported | 60.58% | 56.00 | 0.522 | 0.508 |
| EEmo-Logic + EmoGenie-QA SFT | 77.98% | 68.27 | 0.5140 | 0.5204 |
| EmoProbe (Ours) | 79.96% | 71.11 | 0.6453 | 0.6814 |
Fine-tuning EEmo-Logic on the same EmoGenie-QA training set improves perception and description, confirming the benefit of task-specific training. EmoProbe nevertheless achieves higher performance on both tasks, with a more pronounced advantage on EmoGenie-Assess. Together with the same-backbone ablations in Tab. 6 and the experiment that skips Stage 1, these results support the effectiveness of EmoProbe’s two-stage design, which integrates emotion QA instruction tuning with response-aligned regression for continuous assessment.
Q4 & W6. Human Validation of Automated Description Evaluation.
Comparison with human judgments. We have further conducted a human evaluation on 100 questions selected by stratified random sampling with a fixed seed. The sample contains 90 single-image and 10 image-pair questions, approximating the original test-set proportions, and covers both affective perspectives equally.
We evaluated the strongest general-purpose open-source baseline, the strongest emotion-specialized baseline, the two strongest proprietary models and EmoProbe on the same questions. Their 500 responses were anonymized and presented in randomized order to three annotators for independent scoring. Human evaluation followed the same criteria in App. D.3 as DeepSeek, with mean human ratings combined with the conciseness score into the same equally weighted four-component percentage score. The mean scores and rankings are reported below:
| Model | Human score (0–100) ↑ | DeepSeek score (0–100) ↑ | Human ranking | DeepSeek ranking |
|---|---|---|---|---|
| EmoProbe | 76.58 | 65.43 | 1 | 2 |
| GPT-5.5 | 74.96 | 67.50 | 2 | 1 |
| Qwen3-VL-8B-Instruct | 69.83 | 59.98 | 3 | 4 |
| Claude-Sonnet-4.6 | 69.21 | 61.95 | 4 | 3 |
| EEmo-Logic | 60.83 | 51.90 | 5 | 5 |
Agreement with human judgments. Across the 500 responses, DeepSeek scores correlate with mean human scores at SRCC = 0.839 and PLCC = 0.842. This supports the use of the complete automated scoring protocol for comparative evaluation under the same criteria as the human study.
Consistency of model rankings. Ranking the five models by their mean scores over the shared 100 questions gives a human–DeepSeek Spearman correlation of 0.80. The overall model rankings are broadly consistent, although two adjacent pairs exchange positions. These results support agreement between automated and human evaluation at the model level, while indicating that evaluator choice can affect the ordering of closely performing models.
W1. Contribution of the Benchmark and EmoProbe.
Benchmark contribution. EmoGenie addresses the joint evaluation of expressed and evoked emotions in AI-generated images, bringing together T2I emotion generation and MLLM emotion understanding. The same image collection supports categorical perception, image-pair comparison, evidence-grounded description and continuous VAD/aesthetic assessment. This combination extends the evaluation beyond either synthetic facial expressions or emotion classification alone, as summarized in Tab. 1.
Model contribution. EmoProbe connects QA instruction tuning with continuous assessment through response-aligned, dimension-specific regression. Stage 1 learns categorical and comparative judgments, and Stage 2 maps representations aligned with numerical responses to the corresponding continuous human scores. The same-backbone ablations in Tab. 6 and the additional experiment that applies Stage 2 directly to the pretrained backbone examine the training components and the contribution of Stage 1, respectively. These experiments support the integration of QA instruction tuning and response-aligned regression for unified coarse-to-fine emotion understanding across the three tasks.
We thank the reviewer for the constructive comments. Our point-by-point responses are provided below, which will also be added to the revised paper.
Q1 & W2. Analysis of Prompt Overlap and Generalization.
Image-disjoint and prompt-disjoint splits. The original partition is image-disjoint but not prompt-disjoint. However, shared prompts allow different visual interpretations, as they briefly describe a subject or scene and the intended emotion without specifying its precise depiction. For example, “an elderly woman passenger with a pleased expression, gentle smile” describes expressed emotion, whereas “a scene that evokes awe, a giant cave with towering stalactites” describes evoked emotion. Human labels and ratings are assigned to each generated image, so sharing a prompt does not necessarily imply sharing the same reference answers.
Generalization to unseen prompts. To evaluate performance without shared generation prompts, we have further re-trained both stages after grouping all eight images from each prompt in the same split. We preserved the original 4:1 training/validation-to-test ratio and data sizes, allocating 640 prompts to training/validation and 160 to testing. The test set contains 18,000 QA questions and 2,635 Assess targets:
Model comparison on unseen prompts. We have further evaluated InternVL3.5-8B and EEmo-Logic on this same prompt-disjoint test set. The two baselines retained their original released weights, and EmoProbe was re-trained on the corresponding training partition.
Under the prompt-disjoint partition, EmoProbe shows modest reductions in QA accuracy and assessment correlations, while maintaining broadly comparable performance to that under the original split. EmoProbe also achieves higher QA accuracy and assessment correlations than both baselines on the prompt-disjoint test set. These results support its generalization to unseen prompts and demonstrate that its comparative advantage persists without prompt overlap.