acceptodds
Under review as a conference paper at ICLR 2027

EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

Abstract

Rapid advances in artificial intelligence-generated content (AIGC) have significantly promoted using text-to-image (T2I) models to generate images with specific affective intentions. However, the emotional effects conveyed by generated images frequently deviate from target expectations, and it remains unclear whether multimodal large language models (MLLMs) can reliably interpret the emotion of AI-generated visual content. Existing affective benchmarks are predominantly built on natural images, while affective studies of generated images have largely focused on emotions expressed in portraits, lacking a unified benchmark for studying the emotions expressed and evoked by generated images. We therefore introduce **EmoGenie**, a benchmark of emotion for AI-generated images that evaluates capabilities in T2I generation and MLLM understanding across expressed and evoked emotions. It comprises images generated by eight T2I models using emotion-targeted prompts, annotated with question-answer (QA) pairs, termed **EmoGenie-QA**, and continuous human ratings, termed **EmoGenie-Assess**. EmoGenie-QA contains closed-set questions in single-image and image-pair settings, as well as open-ended questions for emotion description. EmoGenie-Assess benchmarks continuous affective and aesthetic prediction using mean opinion scores (MOSs) aggregated from human ratings in terms of valence, arousal, dominance, and aesthetic quality. We further develop **EmoProbe**, which combines emotion question answering with response-aligned, dimension-specific regression heads for continuous score prediction. Experimental results show that EmoProbe achieves superior results on emotion-related QA tasks, while its affective and aesthetic predictions correlate better with human ratings, demonstrating its state-of-the-art capabilities of unified coarse-to-fine affective understanding for AI-generated images.

Primary area:
datasets and benchmarks
Keywords:
Emotion Benchmark, Text-to-Image Generation, Multimodal Large Language Models, AI-Generated Images

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DsSzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EmoGenie, a benchmark of 6,400 AI-generated images for evaluating expressed and evoked emotions through recognition, comparison, description, and continuous assessment. The authors also propose EmoProbe, which combines emotion instruction tuning with regression heads for continuous affective prediction.

Critical strengths
  1. The paper distinguishes between expressed and evoked emotions in AI-generated images, which is a meaningful direction for evaluating affective image generation and understanding.

  2. EmoGenie provides a comprehensive benchmark covering emotion recognition, image-pair comparison, open-ended description, and continuous affective assessment. The combination of these tasks allows for a more thorough evaluation of MLLM emotion understanding.

  3. The experiments cover 8 T2I models and 14 MLLM baselines, along with ablation studies across different backbones. The results reveal interesting differences in emotion generation and understanding across emotion categories and affective perspectives.

Critical weaknesses
  1. The methodological novelty of EmoProbe is relatively limited. Its two-stage training mainly combines standard instruction tuning with dimension-specific regression heads, without introducing a substantially new approach to emotion understanding.

  2. Although the benchmark contains 6,400 images, they are generated from only 800 prompts. The paper uses an image-disjoint train/test split but does not clearly establish whether prompts are also separated, raising concerns about potential semantic overlap between training and test data.

  3. Evoked emotions are inherently subjective and may vary considerably across individuals. However, the benchmark relies primarily on majority voting and averaged human ratings, which may overlook individual differences and uncertainty in emotional responses.

  4. EmoProbe is specifically trained on EmoGenie, while most baseline MLLMs are evaluated without equivalent task-specific training. This makes it difficult to determine whether the improvements come from the proposed method or simply from training on the benchmark data.

  5. The reliability of human annotations needs further investigation, particularly for continuous dimensions such as arousal and dominance. Reporting human agreement or a noise ceiling would help determine how closely models can reasonably be expected to match human judgments.

  6. The open-ended description task relies on another LLM (DeepSeek-v4-Flash) for evaluation, which may introduce evaluator bias. Additional human evaluation would help validate the reliability of these results.

Questions for authors
  1. The benchmark contains 6,400 images generated from 800 prompts. Does the image-disjoint train/test split also ensure that images generated from the same prompt do not appear in both sets? If not, how might this semantic overlap affect the evaluation results?

  2. Evoked emotions can vary substantially across individuals. Could the authors provide more detailed analysis of human annotation agreement, particularly for emotion categories and continuous VAD ratings? How reliable are the aggregated labels as ground truth?

  3. EmoProbe is trained on EmoGenie, while most baseline MLLMs are evaluated without equivalent task-specific training. Could the authors clarify how much of the performance improvement comes from the proposed architecture versus training on the benchmark data?

  4. The open-ended description task uses DeepSeek-v4-Flash as an evaluator. How reliable are these automated scores compared with human judgments, and could evaluator bias affect the model rankings?

3 comments

We thank the reviewer for the constructive comments. Our point-by-point responses are provided below, which will also be added to the revised paper.

Q1 & W2. Analysis of Prompt Overlap and Generalization.

Image-disjoint and prompt-disjoint splits. The original partition is image-disjoint but not prompt-disjoint. However, shared prompts allow different visual interpretations, as they briefly describe a subject or scene and the intended emotion without specifying its precise depiction. For example, “an elderly woman passenger with a pleased expression, gentle smile” describes expressed emotion, whereas “a scene that evokes awe, a giant cave with towering stalactites” describes evoked emotion. Human labels and ratings are assigned to each generated image, so sharing a prompt does not necessarily imply sharing the same reference answers.

Generalization to unseen prompts. To evaluate performance without shared generation prompts, we have further re-trained both stages after grouping all eight images from each prompt in the same split. We preserved the original 4:1 training/validation-to-test ratio and data sizes, allocating 640 prompts to training/validation and 160 to testing. The test set contains 18,000 QA questions and 2,635 Assess targets:

Data partitionSingle-image QA accuracy (%) ↑Image-pair QA accuracy (%) ↑Overall QA accuracy (%) ↑Assess mean SRCC ↑Assess mean PLCC ↑
Original77.70%81.77%79.96%0.64590.6816
Prompt-disjoint75.73%77.13%76.51%0.60100.6472

Model comparison on unseen prompts. We have further evaluated InternVL3.5-8B and EEmo-Logic on this same prompt-disjoint test set. The two baselines retained their original released weights, and EmoProbe was re-trained on the corresponding training partition.

ModelSingle-image QA accuracy (%) ↑Image-pair QA accuracy (%) ↑Overall QA accuracy (%) ↑Assess mean SRCC ↑Assess mean PLCC ↑
InternVL3.5-8B58.95%63.32%61.38%0.21590.1919
EEmo-Logic50.51%70.08%61.38%0.44390.4691
EmoProbe (Ours)75.73%77.13%76.51%0.60100.6472

Under the prompt-disjoint partition, EmoProbe shows modest reductions in QA accuracy and assessment correlations, while maintaining broadly comparable performance to that under the original split. EmoProbe also achieves higher QA accuracy and assessment correlations than both baselines on the prompt-disjoint test set. These results support its generalization to unseen prompts and demonstrate that its comparative advantage persists without prompt overlap.

Q2 & W3 & W5. Human Annotation Agreement and Aggregated Targets.

Human annotation agreement. We examine agreement for emotion categories and continuous VAD ratings separately. Tab. 7 in App. B.2 reports the coefficients before QA construction. We use Gwet’s AC1 for nominal emotion categories and quadratic-weighted AC2 for the ordered VAD ratings. AC1 is appropriate for categorical judgments without an inherent ordering, whereas AC2 accounts for the magnitude of disagreements between rating levels.

Affective perspectiveAnnotation attributeAgreement measureAgreement coefficient
Expressed emotionEmotion categoryAC10.682
Evoked emotionEmotion categoryAC10.449
Evoked emotionValenceQuadratic-weighted AC20.784
Evoked emotionArousalQuadratic-weighted AC20.798
Evoked emotionDominanceQuadratic-weighted AC20.763

Reliability of aggregated category labels. During QA construction, an emotion-category label is retained only if it receives at least 60% of the votes and has more votes than every other category. We have further quantified the support and stability of these retained labels. Mean support is the average proportion of annotators selecting the retained label for each image. To assess stability, we removed each vote in turn and checked whether the original label remained the sole most-voted category. A tie or a different winning category was counted as a change.

Affective perspectiveMean support for the retained labelLabel preserved after removing one vote
Expressed emotion87.61%94.34%
Evoked emotion78.85%91.42%

The stability calculation checks whether the winning label changes, rather than reapplying the 60% inclusion threshold after vote removal. These results indicate that the retained labels have substantial annotator support and are generally stable to the removal of an individual judgment.

Continuous VAD references. VAD reference scores are obtained by averaging human ratings for each image and dimension. During QA construction, targets with an inter-annotator SD above 2 are excluded to reduce uncertainty arising from substantial rating disagreements. EmoGenie-Assess inherits the retained targets and their corresponding mean scores. Together, the weighted agreement analysis and dimension-specific consistency screening support the use of the retained mean VAD ratings as continuous reference scores for evaluation.

Q3 & W4. Contribution of Task-Specific Training and Model Design.

Contribution of task-specific training. The baseline MLLMs in Tab. 4 are evaluated without additional training on EmoGenie. To examine the contribution of training on the benchmark data, we have further fine-tuned EEmo-Logic on the same EmoGenie-QA training set. Results for perception, description and continuous assessment are reported below:

ModelOverall QA accuracy (%) ↑Description score (0–100) ↑Assess mean SRCC ↑Assess mean PLCC ↑
EEmo-Logic, originally reported60.58%56.000.5220.508
EEmo-Logic + EmoGenie-QA SFT77.98%68.270.51400.5204
EmoProbe (Ours)79.96%71.110.64530.6814

Fine-tuning EEmo-Logic on the same EmoGenie-QA training set improves perception and description, confirming the benefit of task-specific training. EmoProbe nevertheless achieves higher performance on both tasks, with a more pronounced advantage on EmoGenie-Assess. Together with the same-backbone ablations in Tab. 6 and the experiment that skips Stage 1, these results support the effectiveness of EmoProbe’s two-stage design, which integrates emotion QA instruction tuning with response-aligned regression for continuous assessment.

Q4 & W6. Human Validation of Automated Description Evaluation.

Comparison with human judgments. We have further conducted a human evaluation on 100 questions selected by stratified random sampling with a fixed seed. The sample contains 90 single-image and 10 image-pair questions, approximating the original test-set proportions, and covers both affective perspectives equally.

We evaluated the strongest general-purpose open-source baseline, the strongest emotion-specialized baseline, the two strongest proprietary models and EmoProbe on the same questions. Their 500 responses were anonymized and presented in randomized order to three annotators for independent scoring. Human evaluation followed the same criteria in App. D.3 as DeepSeek, with mean human ratings combined with the conciseness score into the same equally weighted four-component percentage score. The mean scores and rankings are reported below:

ModelHuman score (0–100) ↑DeepSeek score (0–100) ↑Human rankingDeepSeek ranking
EmoProbe76.5865.4312
GPT-5.574.9667.5021
Qwen3-VL-8B-Instruct69.8359.9834
Claude-Sonnet-4.669.2161.9543
EEmo-Logic60.8351.9055

Agreement with human judgments. Across the 500 responses, DeepSeek scores correlate with mean human scores at SRCC = 0.839 and PLCC = 0.842. This supports the use of the complete automated scoring protocol for comparative evaluation under the same criteria as the human study.

Consistency of model rankings. Ranking the five models by their mean scores over the shared 100 questions gives a human–DeepSeek Spearman correlation of 0.80. The overall model rankings are broadly consistent, although two adjacent pairs exchange positions. These results support agreement between automated and human evaluation at the model level, while indicating that evaluator choice can affect the ordering of closely performing models.

W1. Contribution of the Benchmark and EmoProbe.

Benchmark contribution. EmoGenie addresses the joint evaluation of expressed and evoked emotions in AI-generated images, bringing together T2I emotion generation and MLLM emotion understanding. The same image collection supports categorical perception, image-pair comparison, evidence-grounded description and continuous VAD/aesthetic assessment. This combination extends the evaluation beyond either synthetic facial expressions or emotion classification alone, as summarized in Tab. 1.

Model contribution. EmoProbe connects QA instruction tuning with continuous assessment through response-aligned, dimension-specific regression. Stage 1 learns categorical and comparative judgments, and Stage 2 maps representations aligned with numerical responses to the corresponding continuous human scores. The same-backbone ablations in Tab. 6 and the additional experiment that applies Stage 2 directly to the pretrained backbone examine the training components and the contribution of Stage 1, respectively. These experiments support the integration of QA instruction tuning and response-aligned regression for unified coarse-to-fine emotion understanding across the three tasks.

Reviewer_vKQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EmoGenie, a benchmark for evaluating expressed and evoked emotions in AI-generated images, distinguishing the emotions displayed by depicted individuals from those elicited in viewers. The benchmark contains 6,400 images generated by eight text-to-image models using 800 prompts, with 15 independent human annotations per image. These annotations support two subsets: EmoGenie-QA for emotion question answering and EmoGenie-Assess for continuous affective and aesthetic assessment. The authors also propose EmoProbe, built on InternVL3.5-8B-Instruct through two-stage training: emotion instruction tuning followed by continuous assessment with response-aligned regression heads. Experiments evaluate the target-emotion alignment of text-to-image models and the emotion-understanding capabilities of multimodal large language models.

Critical strengths
  • Jointly evaluating expressed and evoked emotions provides a useful, broader assessment. The emotion displayed by a depicted individual does not necessarily match the emotion elicited in a viewer. EmoGenie covers both perspectives through single-image judgments, image-pair comparisons, open-ended descriptions, and continuous scoring, offering a more comprehensive evaluation than emotion classification alone.

  • The paper is clearly presented and reveals informative differences across models and tasks. The motivation, dataset construction, model design, and experimental results are well organized and easy to follow.

Critical weaknesses
  • The selection process for continuous scores is insufficiently explained. Under the annotation setup in Section 3.2, all 3,200 evoked-emotion images receive ratings on three VAD dimensions, and all 6,400 images receive aesthetic ratings. This should yield 3,200 × 3 + 6,400 = 16,000 image–dimension mean scores. However, EmoGenie-Assess contains only 13,175 scores.
  • The need for two-stage training and the choice of loss weights are not sufficiently validated. The authors report the main training hyperparameters, but Appendix C explains the choice of λ_{VAD} = λ_{aes} = 1.5 only as balancing continuous-score regression with textual response supervision. The paper does not describe how this value was selected or report comparisons with alternative values.
Questions for authors
  • Data selection. The stated annotation setup should yield 16,000 continuous scores. Why were only 13,175 included in EmoGenie-Assess? What were the inclusion and exclusion criteria, and why do the retained counts differ across dimensions? Did selection substantially change the emotion-category, generator, or score distributions?

  • Training settings and loss weights. Does Stage 2 use the same batch size, warm-up ratio, learning-rate schedule, and weight decay as Stage 1? How was λ_{VAD} = λ_{aes} = 1.5 selected and validated? Are the main results stable under alternative values?

  • Contribution of Stage 1. Have the authors evaluated skipping Stage 1 and applying the Stage 2 training procedure directly to the pretrained backbone? A comparison using identical Stage 2 data, training settings, and test sets, with results for perception, description, and continuous assessment, would help establish the contribution of Stage 1.

  • Data and code availability. Could the authors provide anonymous access during review to sample data and the main code for data processing, training, and evaluation, so that the above procedures can be verified?

2 comments

We thank the reviewer for the constructive comments. Our point-by-point responses are provided below, which will also be added to the revised paper.

Q1 & W1. Inclusion Criteria and Analysis of Selection Effects.

Inclusion and exclusion criteria. EmoGenie-Assess retains the images and rating dimensions selected for numerical-rating questions in EmoGenie-QA, together with their corresponding continuous human scores. This correspondence supports coarse-to-fine learning, from discrete rating judgments to continuous-score prediction for the same images and dimensions.

Selection takes place in two steps during QA construction. First, we exclude image-dimension targets whose inter-annotator standard deviation exceeds 2 on the original rating scale. This consistency screening removes 1,281 of the 16,000 applicable targets, leaving 14,719 eligible scores. Second, we sample 40,000 single-image questions from 44,828 eligible questions using dimension-proportional quotas and retaining paired question formats together. This QA sampling leaves a further 1,544 eligible image-dimension targets unselected.

The resulting EmoGenie-Assess subset contains 13,175 scores: 2,590 valence, 2,475 arousal, 2,400 dominance and 5,710 aesthetic scores. Differences in annotation agreement and sampling quotas across dimensions account for the unequal retained counts.

Effect of selection on data distributions. We have further compared all 16,000 applicable image-dimension targets with the 13,175 retained targets. Percentages were calculated over these targets, with emotion categories summarized within each affective perspective and generators summarized across both perspectives.

Emotion-category distribution. Categories refer to the target emotion in the generation prompt. Expressed and evoked emotions are reported separately below.

Expressed emotion.

Selection stageTargets (n)AngerDisgustFearHappinessNeutralSadnessSurprise
Before selection3,20013.25%15.00%15.00%11.75%11.00%16.50%17.50%
After selection2,85512.92%14.99%14.89%11.49%11.38%16.85%17.48%

Evoked emotion.

Selection stageTargets (n)AmusementAngerAweContentmentDisgustExcitementFearSadness
Before selection12,80012.50%12.50%12.50%12.50%12.50%12.50%12.50%12.50%
After selection10,32013.29%12.32%11.43%11.93%12.03%13.00%12.83%13.17%

Generator distribution.

Selection stageTargets (n)FLUX.2GPT-Image-2PixArt-ΣQwen-ImageSD3SD3.5Seedream-5.0Z-Image
Before selection16,00012.50%12.50%12.50%12.50%12.50%12.50%12.50%12.50%
After selection13,17512.47%12.35%12.64%12.64%12.55%12.46%12.39%12.52%

Score statistics. We report the mean ± population standard deviation of image-level mean ratings on each dimension’s original scale.

Selection stageValence (1–9)Arousal (1–9)Dominance (1–9)Aesthetic quality (1–5)
Before selection5.103 ± 1.5775.868 ± 0.8545.162 ± 0.9203.298 ± 0.685
After selection5.149 ± 1.6245.933 ± 0.8595.219 ± 0.9253.297 ± 0.686

Emotion-category and generator proportions remain similar after selection, as do the means and standard deviations of the continuous scores. These summaries show no substantial shift in the measured composition or score statistics of the retained targets.

Q2 & W2. Training Configurations and Loss-Weight Sensitivity.

Training settings. Both stages use an effective batch size of 16, cosine learning-rate decay, weight decay of 0.1 and one training epoch. The learning rate and warm-up ratio differ as listed below. Stage 2 continues from the Stage 1-adapted backbone with a lower learning rate and no warm-up.

Training parameterStage 1Stage 2
Effective batch size1616
Learning rate1 × 10⁻⁴5 × 10⁻⁵
Warm-up ratio0.030
Learning-rate scheduleCosineCosine
Weight decay0.10.1
Epochs11

Selection of the loss weights. In Eq. (4) of Sec. 4.2, each assessment example provides a target for one rating dimension, and the masks activate only the corresponding regression term:

L=Ltext+λVAD∑d∈DVADmdLd+λaesmaesLaes.\mathcal{L}=\mathcal{L}_{\mathrm{text}}+\lambda_{\mathrm{VAD}}\sum_{d\in \mathcal{D}_{\mathrm{VAD}}} m_{d}\mathcal{L}_{d}+\lambda_{\mathrm{aes}}m_{\mathrm{aes}}\mathcal{L}_{\mathrm{aes}}.

The shared coefficient controls the contribution of the active regression loss relative to text supervision. After normalizing continuous targets to [0,1], our initial observations showed a smaller regression MSE than text cross-entropy. We therefore used a coefficient above 1 to increase the contribution of continuous-score supervision, adopting λVAD=λaes=1.5\lambda_{\mathrm{VAD}}=\lambda_{\mathrm{aes}}=1.5 as a moderate empirical choice. The additional experiments below examine both this choice and sensitivity to alternative values.

Stability under alternative loss weights. We have further trained with shared coefficients of 1.0 and 2.0, keeping the Stage 2 data and evaluation sets unchanged, and compared them with the original 1.5 setting:

Shared loss weight (λ)Assess mean SRCC ↑Assess mean PLCC ↑QA accuracy (%) ↑
1.00.64230.677980.10%
1.50.64530.681479.96%
2.00.63730.675180.07%

The original setting of 1.5 achieves the highest mean SRCC and PLCC among the tested values while maintaining comparable QA accuracy. Performance also remains similar at 1.0 and 2.0, supporting both the chosen coefficient and the stability of the results within this range.

Q3 & W2. Evaluation of the Contribution of Stage 1.

We have further evaluated skipping Stage 1 and applying the Stage 2 training procedure directly to the pretrained InternVL3.5-8B backbone. The regression-head architecture, test sets and inference protocol are the same as those used for EmoProbe. We report results for perception, description and continuous assessment below:

Training configurationQA accuracy (Single) (%) ↑QA accuracy (Pair) (%) ↑Overall QA accuracy (%) ↑Description (0–100) ↑Assess mean SRCC ↑Assess mean PLCC ↑
Stage 2 only60.74%72.65%67.36%69.190.63960.6741
EmoProbe (Ours)77.70%81.77%79.96%71.110.64530.6814

Contribution of Stage 1. EmoProbe performs better across perception, description and continuous assessment than training Stage 2 directly on the pretrained backbone. Stage 1 develops the categorical and comparative emotion understanding required by the QA tasks. It also provides an initialization learned from discrete rating judgments before Stage 2 optimizes the corresponding continuous scores. The results support this coarse-to-fine training design, with the principal benefit observed in perception.

Q4. Anonymous Access to Data and Code.

Thank you for the suggestion. We agree that access to data and code is important for reproducibility. Anonymous access to the benchmark test files, example images, and training and evaluation code is available at the link below.

Anonymous link: Anonymous repository.

Reviewer_PchROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this work, the authors introduce Emogenie, which is a benchmark used for the evaluation of emotions expressed in synthetic images. The data contains 6400 images which are generated by 8 T2I models by using 800 prompts with 15 human annotations/image. The data also supports question answering, open-ended descriptions, and continuous prediction of valence, arousal, dominance, and aesthetic quality. In addition to the Emogenie, the authors also develop Emoprobe, which is a MLLM trained via emotion instrcution tuning followed by continuous assessment with dimension-specific regression heads. The experiments show improvements in question-answering accuracy and correlations with human ratings.

Critical strengths
  • I like that expressed emotions from evoked emotions are distinguished. The combination of generation evaluation with understanding of emotions makes the data relevant for both T21 and emotion understanding.
  • Each image has 15 independent judgments, which is a huge human effort.
  • The data covers single-image judgments, image comparisons, descriptions, and continuous assessment. The results show the difficulties across various categories, including weak evoked-anger alignment or challenging arousal prediction.
  • Emoprobe improves scores across all four dimensions. Ablations on two backbones provide evidence that the second training stage helps.
Critical weaknesses
  • The comparison largely contrast a supervised regressor with models evaluated through their logits. Table 6 also changes both training and prediction approaches when adding Stage 2. So, the results show the value of supervised adaptation more clearly than the specific value of the proposed token representation. I think comparisons with supervised pooled-feature regressors are needed.
  • The judge receives reference text but no image, right? So it cannot independently verify visual claims or recognize valid interpretations missing from the reference. Also, a length-based conciseness score contributes one quarter of the final score. Judgment score with image provided would be better.
  • Emotion evoked is subjective, so different responses can be legitimate. A distribution-aware evaluation, with annotation uncertainty, and information about annotators’ cultural backgrounds(if possible, not necessary as it can violate ethical grounds) would be an interesting analysis strengthining the paper.
  • EmoProbe gets two advantages at once. It is trained on human emotion scores and It uses a special component to predict those scores. For the other models, their scores are inferred from how strongly they prefer emotional words such as “calm” versus “excited.” So, I am not sure how much of the gains reported are from the design or the data.
Questions for authors
  • For 3200 evoked images for three VAD dimensions, plus aesthetics for all 6400 images, would get 16000 image dimension scores. But in appendix B.4, it says that 13175, with different counts for VAD dimension. Can the authors clarify this?
  • The splits are image-disjoint, but can the authors clarify if a shared image partition is enforced across all tasks, and whether images generated from the same prompt remain together. A prompt-disjoint evaluation can further establish generalization to unseen prompts.
4 comments

We thank the reviewer for the constructive comments. Our point-by-point responses are provided below, which will also be added to the revised paper.

Q1. Clarification of the Construction of EmoGenie-Assess.

EmoGenie-Assess retains the images and rating dimensions selected for numerical-rating questions in EmoGenie-QA, together with their corresponding continuous human scores. This correspondence supports coarse-to-fine learning: the model first learns discrete rating judgments through QA instruction tuning, then learns to predict continuous scores for the same images and dimensions.

The 3,200 evoked-emotion images provide 9,600 VAD scores, and all 6,400 images provide aesthetic scores, giving 16,000 image-dimension scores before selection. The final count follows from two steps during QA construction:

Consistency screening. For each image, we assess agreement separately for each rating dimension and exclude scores whose inter-annotator standard deviation exceeds 2 on the original scale. This removes 1,281 image-dimension targets, leaving 14,719 eligible continuous scores.

QA sampling. To obtain a fixed-size QA set, we sample 40,000 single-image questions from 44,828 eligible questions using dimension-proportional quotas and retaining paired question formats together. This sampling leaves 1,544 eligible image-dimension targets outside the final QA set. EmoGenie-Assess inherits the remaining 13,175 targets and their continuous human scores.

Rating dimensionBefore selection (n)Excluded by SD > 2 (n)Not selected in QA sampling (n)Retained in EmoGenie-Assess (n)
Valence3,2002843262,590
Arousal3,2004692562,475
Dominance3,2005282722,400
Aesthetic quality6,40006905,710
Total16,0001,2811,54413,175

Together, consistency screening and QA sampling reduce the initial 16,000 scores to 13,175. Differences in annotation agreement and sampling quotas across dimensions account for the unequal retained counts.

Q2. Shared Data Partitions and Prompt-Disjoint Evaluation.

Shared image partition across all tasks. We enforce the same image-disjoint partition across all tasks, so no image appears in both training and testing.

The original partition is image-disjoint, but images generated from the same prompt may occur in different splits. As described in App. A, our prompts briefly describe a subject or scene and the intended emotion without prescribing exactly how these should be depicted. Examples include “an elderly woman passenger with a pleased expression, gentle smile” for expressed emotion and “a scene that evokes awe, a giant cave with towering stalactites” for evoked emotion. Different generators can thus produce distinct visual interpretations of the same prompt. Human labels and ratings are assigned to each generated image, so sharing a prompt does not necessarily imply sharing the same reference answers.

Prompt-disjoint evaluation. To examine generalization to unseen prompts, we have further re-trained both stages with all eight images generated from each prompt assigned to the same split. Following the original 4:1 ratio, 640 prompts were allocated to training/validation and 160 to testing. We retained the original data sizes: 72,000 QA questions and 10,540 Assess targets for training/validation, and 18,000 QA questions and 2,635 Assess targets for testing. The QA test set contains 8,000 single-image and 10,000 image-pair questions, with question formats aligned to training.

Data partitionSingle-image QA accuracy (%) ↑Image-pair QA accuracy (%) ↑Overall QA accuracy (%) ↑Assess mean SRCC ↑Assess mean PLCC ↑
Original77.70%81.77%79.96%0.64590.6816
Prompt-disjoint75.73%77.13%76.51%0.60100.6472

Comparison on the same prompt-disjoint test set. We have further evaluated InternVL3.5-8B and EEmo-Logic on the same test questions and assessment targets. Both baselines were evaluated with their original released weights, while EmoProbe was re-trained on the prompt-disjoint training partition.

ModelSingle-image QA accuracy (%) ↑Image-pair QA accuracy (%) ↑Overall QA accuracy (%) ↑Assess mean SRCC ↑Assess mean PLCC ↑
InternVL3.5-8B58.95%63.32%61.38%0.21590.1919
EEmo-Logic50.51%70.08%61.38%0.44390.4691
EmoProbe (Ours)75.73%77.13%76.51%0.60100.6472

EmoProbe shows modest reductions in QA accuracy and assessment correlations under the prompt-disjoint partition, while maintaining broadly comparable performance to that under the original split. It also outperforms both baselines on perception and continuous assessment on the prompt-disjoint test set. These results complement the original image-disjoint evaluation and support the effectiveness of EmoProbe on images generated from unseen prompts.

W1. Token Representation and Supervised Pooled-Feature Regression.

Numerical-response token representation. We use the final-layer hidden state at the numerical-response position as the input to the regression head. This representation integrates the image, the rating instruction and the preceding response context, associating continuous-score prediction with the judgment expressed in the response.

Supervised pooled-feature comparison. We have further trained a pooled-feature regressor under the same Stage 2 supervision and training settings, replacing the numerical-response token representation with mean-pooled multimodal input features. Both models were evaluated on the same QA and assessment test sets:

Regression input representationSingle-image QA accuracy (%) ↑Image-pair QA accuracy (%) ↑Overall QA accuracy (%) ↑Assess mean SRCC ↑Assess mean PLCC ↑
Mean-pooled77.03%82.19%79.89%0.64850.6741
Token representation (Ours)77.70%81.77%79.96%0.64590.6816

The dimension-specific correlations are reported below; each entry gives SRCC / PLCC:

Regression input representationValence ↑Arousal ↑Dominance ↑Aesthetic quality ↑
Mean-pooled0.8321 / 0.83980.4757 / 0.48260.6256 / 0.61570.6606 / 0.7582
Token representation (Ours)0.8367 / 0.85510.4811 / 0.49820.6284 / 0.61560.6374 / 0.7574

The two representations achieve comparable overall QA accuracy. The numerical-response token representation yields slightly higher SRCC across all three VAD dimensions and higher PLCC for valence and arousal, with virtually identical PLCC for dominance. Mean pooling achieves higher aesthetic SRCC, while aesthetic PLCC is similar. This pattern may reflect the benefit of reading out a representation associated with a specific affective judgment for VAD prediction, while aesthetic assessment may benefit more from aggregating information across input positions. The proposed token representation thus improves agreement with human VAD rankings and increases the overall mean PLCC while maintaining overall QA performance.

W2. Visual Grounding and Conciseness in Description Evaluation.

Evaluator inputs. As specified in App. D.3, DeepSeek receives the question, candidate response, human reference answer and scoring rubric, without the image or model identity.

Evaluation design and human validation. We use human reference answers to assess response quality against a common interpretation of the image. An image-conditioned evaluator would additionally rely on its own visual emotion understanding when judging the response. Reference-based scoring avoids this additional dependence, although it cannot independently validate visual interpretations absent from the reference.

To validate the resulting scores against human judgments, we have further conducted a study on 100 questions selected through stratified random sampling: 90 single-image and 10 image-pair questions, with equal coverage of expressed and evoked emotions. Three annotators independently scored 500 anonymized responses from five models on the same questions. The images were provided to the annotators as auxiliary context, and the scoring criteria were consistent with App. D.3. The complete scores and model rankings are reported below:

ModelHuman score (0–100) ↑DeepSeek score (0–100) ↑Human rankingDeepSeek ranking
EmoProbe76.5865.4312
GPT-5.574.9667.5021
Qwen3-VL-8B-Instruct69.8359.9834
Claude-Sonnet-4.669.2161.9543
EEmo-Logic60.8351.9055

The automated scores correlate with mean human scores at SRCC = 0.839 and PLCC = 0.842. Ranking the models by their mean scores yields a human–DeepSeek Spearman correlation of 0.80. These results show broadly consistent model rankings, with differences in the ordering of closely performing models, and support the use of the reference-based protocol for comparative evaluation.

Role of conciseness. Conciseness is included to counter verbosity bias, whereby longer answers may receive higher completeness scores without providing more useful information. We assign equal weight to correctness, completeness, relevance and conciseness to balance informative coverage with concise expression.

W3 & Ethics. Individual Differences and the Annotator Population.

Distribution of individual annotations. We have further examined the original category votes for all 3,200 expressed-emotion and 3,200 evoked-emotion images, before QA selection. For each image, we calculated the proportion of votes assigned to its most-selected category and whether all annotators selected the same category:

Affective perspectiveImages (n)Mean support for the most-selected categoryImages with unanimous category votes
Expressed emotion3,20083.11%48.38%
Evoked emotion3,20069.93%21.06%

Evoked-emotion votes are more dispersed than expressed-emotion votes, consistent with greater individual variation in emotional responses to an image. For the 3,200 evoked-emotion images, the mean within-image SD is 1.283 for valence, 1.407 for arousal and 1.463 for dominance on the 1–9 scale. These statistics describe the variation among annotators underlying the aggregated labels and scores.

Annotator recruitment and preparation. Annotators were recruited from students at our university and received detailed annotation instructions and a brief training session with illustrative examples. This preparation provides a shared understanding of the emotion categories and rating scales. Majority voting and averaging then combine multiple judgments into the evaluation references. The recruited panel defines the population represented by these references; cross-cultural differences were not evaluated.

W4. Contribution of Supervision and the Regression Design.

Effect of training on the benchmark data. To examine the contribution of task-specific training, we have further fine-tuned EEmo-Logic on the same EmoGenie-QA training set. This provides a baseline adapted to the benchmark’s QA supervision, rather than relying solely on comparisons with models that have not received such training. The results are reported below:

ModelQA accuracy (%) ↑Description (0–100) ↑Assess mean SRCC ↑Assess mean PLCC ↑
EEmo-Logic, originally reported60.58%56.000.5220.508
EEmo-Logic + EmoGenie-QA SFT77.98%68.270.51400.5204
EmoProbe (Ours)79.96%71.110.64530.6814

Training on EmoGenie-QA improves EEmo-Logic’s perception and description performance, confirming the benefit of benchmark-specific supervision. EmoProbe retains higher performance on both tasks and a larger advantage on continuous assessment. Its Stage 2 directly optimizes continuous-score prediction with human ratings, whereas the QA-adapted baseline is trained on discrete QA responses. Together with the same-backbone ablations in Tab. 6, these results support combining instruction tuning with continuous-score supervision and response-aligned regression.

Comments on the paper

1 comment

Summary of Our Responses and Additional Experiments

We sincerely thank the reviewers for their careful assessment and constructive feedback. We have further conducted experiments and annotation analyses to examine benchmark reliability, generalization to unseen prompts and the contributions of EmoProbe’s training design. The main additions and clarifications are summarized below.

  1. Benchmark construction and selection effects. We clarified how consistency screening and QA sampling reduce the initial 16,000 image-dimension scores to the 13,175 scores in EmoGenie-Assess. We also explained the correspondence between the discrete QA targets and continuous assessment scores. Comparisons before and after selection show that emotion-category and generator proportions, together with continuous-score means and standard deviations, remain largely unchanged.

  2. Generalization to unseen prompts. We have further re-trained both stages under a prompt-disjoint partition, preserving the original split ratio and data sizes. We also evaluated InternVL3.5-8B and EEmo-Logic on the resulting test set. EmoProbe achieved 76.51% overall QA accuracy and mean assessment SRCC/PLCC of 0.6010/0.6472, outperforming both baselines on perception and continuous assessment. These results extend the original image-disjoint evaluation and demonstrate that EmoProbe’s comparative advantage persists without prompt overlap.

  3. Training and representation comparisons. We have further evaluated applying Stage 2 directly to the pretrained backbone. The complete EmoProbe model performed better on perception, description and continuous assessment, supporting the contribution of preceding QA instruction tuning. We additionally trained a supervised pooled-feature regressor under the same Stage 2 supervision and training settings. Overall QA accuracy was comparable; the numerical-response token representation yielded slightly higher SRCC across all three VAD dimensions, whereas mean pooling performed better on aesthetic quality. Fine-tuning EEmo-Logic on EmoGenie-QA also improved its perception and description scores, while EmoProbe retained higher performance across all three tasks.

  4. Loss-weight sensitivity. We have further evaluated shared regression-loss weights of 1.0 and 2.0 alongside the original 1.5 setting. The original setting achieved the highest mean assessment SRCC and PLCC, while results remained similar across the tested values. This comparison supports both the selected coefficient and stability within the evaluated range.

  5. Human annotation and evaluator reliability. We have further analyzed annotation variability and the support and stability of aggregated category labels. We also conducted a human validation study covering 500 responses from five models on 100 shared, stratified-randomly sampled questions. Three annotators followed the same scoring criteria as the automated evaluator. Automated scores correlated with mean human scores at SRCC = 0.839 and PLCC = 0.842; the correlation between the five-model overall rankings was 0.80. The rankings were broadly consistent, with differences between closely performing models.

  6. Anonymous access. Benchmark test files, example images, and training and evaluation code are available in our anonymous repository.

Detailed experimental results and methodological clarifications are provided in the reviewer-specific responses. We appreciate the opportunity to strengthen the evidence and clarify the benchmark and model design, and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.