- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces NeuroAes, a neuroaesthetics-grounded dataset and benchmark for photographic aesthetic understanding, together with AesMind, a specialized multimodal large language model. NeuroAes contains 25K images with fine-grained annotations spanning sensory-motor attributes, aesthetic and evoked emotions, and structured professional critiques, including human-rated continuous VAD scores. Based on this dataset, AesMind is trained through a three-stage pipeline consisting of skill harnessing, supervised fine-tuning, and GRPO with task-specific rewards. Experiments show strong performance on the proposed benchmark and competitive cross-dataset generalization, particularly for fine-grained perception, continuous aesthetic assessment, and structured photographic critique generation.
Critical strengths
-
The paper presents a solid and well-motivated neuroaesthetics-grounded taxonomy that goes beyond a single aesthetic score and organizes photographic assessment into perception, emotion, and critique.
-
NeuroAes is a substantial and carefully constructed dataset, with 25K images, fine-grained labels, human-rated VAD annotations, and a dedicated 5K-image benchmark covering perception, assessment, and description tasks.
-
The experimental evaluation is comprehensive and generally convincing, including strong results on the proposed benchmark, ablations of the three training stages, and cross-dataset evaluation on several existing aesthetic benchmarks.
Critical weaknesses
- The annotation pipeline mixes human judgments with Gemini-generated annotations, but the paper does not provide sufficient calibration between these two sources. It would be useful to report how much of the training data is purely human-annotated, how well Gemini annotations agree with human judgments, and ablations comparing training with human-only, LLM-only, and combined annotations.
Questions for authors
-
Could the authors provide more details about the human annotators’ backgrounds, especially the distribution of their artistic and photographic expertise? I am particularly interested in whether these annotators can truly be considered experts, given that the paper emphasizes professional-level aesthetic assessment.
-
Since much of the evaluation is conducted within the NeuroAes taxonomy and annotation framework, I am curious how AesMind would perform on independent expert-grounded benchmarks. In particular, Visual Aesthetic Benchmark [1] is aligned with expert consensus on aesthetic preferences, and AesBench [2] provides another external benchmark for expert-level aesthetic assessment. Results on these benchmarks would make the generalization claims more convincing.
-
The annotation pipeline combines human judgments and Gemini-generated annotations, but it is unclear how well these two sources are calibrated. Could the authors report how much of the training data is purely human-annotated, the agreement between human and Gemini annotations, and ablations comparing training on human-only, LLM-only, and combined annotations?
[1] https://arxiv.org/abs/2605.12684 [2] https://arxiv.org/abs/2401.08276
6 comments
1. W1 & Q3: Annotation source and human calibration (continued)
(b) Human verification and Gemini annotation quality
To assess the quality of Gemini-generated annotations, we analyzed the human-review logs, which reveal varying levels of human intervention across annotation dimensions. Across 41 fields, we identified 77,476 unique image–field records with logged human interventions among 971,305 post-review entries, corresponding to an overall intervention rate of 7.976%. The following table (Table 13 in the revised paper) illustrates substantial differences in the extent of human review across annotation dimensions, particularly between fine-grained photographic and affective attributes and basic semantic attributes.
| Annotation field | Format | Human-review records | Post-review entries | Intervention rate |
|---|---|---|---|---|
| Additional composition patterns | Multiple-choice | 18,228 | 23,731 | 76.811% |
| Portrait composition | Multiple-choice | 6,795 | 12,298 | 55.253% |
| Color-quality evaluation | Single-choice | 11,169 | 25,000 | 44.676% |
| Portrait angle | Multiple-choice | 3,773 | 12,291 | 30.697% |
| Discrete emotion label | Single-choice | 6,378 | 25,000 | 25.512% |
| Number of people | Single-choice | 157 | 25,000 | 0.628% |
| Main subject | Open-ended | 164 | 25,000 | 0.656% |
| Animal presence | Single-choice | 22 | 25,000 | 0.088% |
The results reveal clear dimension-wise differences in Gemini annotation quality, as reflected by the extent of subsequent human intervention. Even among single-choice fields, color-quality evaluation and discrete emotion labels exhibit substantially higher intervention rates than basic semantic attributes such as people counting and animal presence. This suggests that Gemini provides more reliable preliminary annotations for basic semantic recognition, whereas fine-grained aesthetic and affective attributes require substantially more human verification. The high intervention rates for composition patterns and portrait framing further highlight the difficulty of accurately annotating complex photographic properties. Although intervention rates may also be influenced by annotation format and field applicability, they provide empirical evidence that human verification plays a particularly important role in refining Gemini-generated annotations for subjective and fine-grained aesthetic dimensions.
(c) Human-only, LLM-only, and combined training
Thank you for raising this important question regarding the contribution of different annotation sources to model training. A controlled comparison of human-only, LLM-only, and combined training is challenging due to the task-specific annotation protocols and the resulting lack of comparable training targets across sources.
Specifically, for Perception and Description, annotations are initially generated by Gemini and subsequently refined or verified by human annotators. Thus, independently human-authored annotations are not available for these tasks. Moreover, directly using raw Gemini-generated annotations without human refinement would introduce substantial annotation errors, making it difficult to establish a meaningful LLM-only training baseline. Conversely, Assessment relies entirely on human ratings to derive continuous VAD targets, and no corresponding LLM-generated targets are available for an LLM-only comparison. Therefore, the three training configurations cannot be consistently constructed across all tasks without changing the task composition and supervision targets.
2. Q1: Participant backgrounds and annotation training
Thank you for asking for a clearer account of the annotators' artistic and photographic experience. All 71 participants took part in the subjective rating experiment, and we collected background questionnaires and Big Five personality measures from each of them. We additionally invited a subset with photographic experience to perform human label verification. The table below (Table 11 in the revised paper) summarizes the full rating cohort, rather than the verification subset alone.
(a) Background composition
| Background | Category | N | Percentage |
|---|---|---|---|
| Gender | Male | 35 | 49.3% |
| Gender | Female | 36 | 50.7% |
| Education | Undergraduate | 26 | 36.6% |
| Education | Postgraduate | 45 | 63.4% |
| Artistic experience | Novice | 29 | 40.8% |
| Artistic experience | Intermediate | 30 | 42.3% |
| Artistic experience | Advanced | 12 | 16.9% |
| Photographic experience | Novice | 24 | 33.8% |
| Photographic experience | Intermediate | 36 | 50.7% |
| Photographic experience | Advanced | 11 | 15.5% |
Across the rating cohort, 66.2% reported intermediate or advanced photographic experience, and 59.2% reported intermediate or advanced artistic experience. These are questionnaire-based experience levels rather than verified professional credentials. For label verification, participants were selected from those with photographic experience. The background and personality questionnaires document participant variation, rather than serving as a substitute for task-specific preparation.
(b) Standardized training and field-specific guidance
All 71 rating participants received standardized training, irrespective of their prior experience. The verification subset additionally received a detailed manual for each annotation field. These manuals define the dimension and its response options, illustrate the options with multiple photographic examples, and explain ambiguous or difficult cases. This combines the verifiers' photographic experience with a common operational basis for checking labels. As reported in our first response, the review logs further document substantial human intervention in fields such as color-quality evaluation and portrait composition.
In summary, the rating cohort includes a majority with photographic experience, while label verification is performed by a selected photography-experienced subset supported by standardized training and detailed field-specific guidance. We have included the participant backgrounds and Big Five summary statistics in the supplementary material, with the updates highlighted in teal.
To Reviewer D7zv:
Thank you for the constructive and detailed feedback. We address these points below and will incorporate the corresponding analyses and clarifications into the revised manuscript.
1. W1 & Q3: Annotation source and human calibration
Thank you for highlighting the need to clarify annotation source and quantify the role of human verification. NeuroAes contains three complementary annotation forms corresponding to the Assessment, Perception, and Description tasks in NeuroAes-Bench. Human judgments and Gemini outputs play different roles in their construction, rather than being mixed together as interchangeable labels pooled for the same task.
(a) Annotation sources and training-data composition
As illustrated in Step 2 of Figure 2, three tasks have different annotation processes. (1) For assessment task, the six continuous VAD dimensions—valence, arousal, and dominance for aesthetic and evoked emotions—are entirely human-rated. Each image is rated by 15 participants, and the ratings are standardized and aggregated into MOS. Gemini does not generate or replace these scores. (2) For perception task, Gemini produces preliminary annotations that are subsequently checked and corrected by human annotators. Specially, labels of the emotion-valuation system are additionally grounded in the human VAD scores. (3) For description task, the existing human-written comments in the image metadata serve as the primary reference, with the image and human-verified labels providing supplementary grounding for Gemini's structured generation. These critiques are therefore human-grounded but Gemini-generated, not independently human-authored. The annotation types, data quantities, and corresponding sources for the three tasks are summarized in the table below and in Table 15 in the revised paper.
As shown in the table below and in Table 16 in the revised paper, 11.75% of the training QA examples are purely human-annotated, with continuous VAD targets derived exclusively from human ratings. The remaining 88.25% involve human contributions through verification, correction, or reference annotations, combined with Gemini-generated content.