- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CHIAA-Bench, containing 20.8k expert-annotated images, with an assessment protocol covering four major dimensions and ten fine-grained attributes. It further proposes CineAesNet, a lightweight EVA02-based aesthetic predictor. Experiments show strong performance against existing aesthetic models and MLLMs on both in-domain and cross-source evaluations. The authors also investigate model-assisted annotation and the adaptation of MLLMs to this task.
Critical strengths
-
Existing aesthetic benchmarks do not adequately capture the characteristics of cinematic human images. CHIAA-Bench addresses this gap with a well-motivated evaluation protocol and expert-refined annotations. I consider this the main contribution of the paper.
-
The authors compare against various aesthetic models and MLLMs, evaluate cross-source generalization, and investigate the contributions of model-generated scores and human annotations. These experiments provide useful evidence for the value of the proposed dataset.
Critical weaknesses
-
CineAesNet essentially combines a pretrained EVA02 backbone with a lightweight prediction head and conventional training objectives. It is unclear how much of the improvement comes from the model design rather than the new training data. The architectural contribution appears relatively limited.
-
The authors demonstrate that fine-tuned 4B and 9B MLLMs already achieve performance close to CineAesNet. However, the fine-tuning configuration, particularly whether the visual encoder is updated, is not sufficiently clear. More controlled comparisons with smaller MLLMs and different visual adaptation strategies would help establish the advantage of CineAesNet.
Questions for authors
-
When fine-tuning the 4B, 9B, and 27B MLLMs, is the visual encoder frozen? Which modules are updated through LoRA, and what are the corresponding training configurations?
-
Can the authors provide inference latency, throughput, and memory comparisons between CineAesNet and the fine-tuned smaller MLLMs under comparable settings? This would better substantiate the claimed efficiency advantage.
4 comments
Post rebuttal
I appreciate the authors' detailed rebuttal. The clarified MLLM training configurations and additional efficiency comparisons adequately address my original questions. The supplementary analyses also strengthen the experimental evidence.
However, I agree with Reviewers Qvt6 and cie3 regarding the limited technical novelty. CineAesNet relies on a standard backbone and conventional training objectives, without meaningful designs tailored to cinematic human aesthetics. Moreover, the distinction between CHIAA and existing human image aesthetic assessment remains insufficiently justified.
While I recognize the value of the benchmark and its annotations, these fundamental concerns remain unresolved. I therefore lower my rating from to 2.
Re-response to Reviewer Gn4t
We thank the reviewer for confirming that our responses addressed the original questions. We would like to clarify the two remaining concerns.
1) Technical novelty.
-
On the one hand, the novelty of this work lies in task formulation, benchmark construction, and annotation protocol design. To the best of our knowledge, research in the field of human aesthetic assessment remains relatively scarce, and studies specifically targeting the cinematic domain are virtually nonexistent. We believe that contributing a benchmark based on lightweight, task-specific models is a well-established research paradigm. For this task, we designed a ten-dimensional assessment protocol, automated human-image filtering (Appendix A.1), and explicit expert grading criteria to construct a usable cinematic human aesthetic benchmark.
-
On the other hand, we have not neglected the design for cinematic human image aesthetic assessment. Our task-specific designs also include score fusion across MLLMs, aesthetic models, and quantitative visual cues (Appendices A.3–A.4). For MLLM adaptation, we also adopt a well-justified architecture, we added an extra Regression Head and a Classification Head specifically tailored to this task. We additionally report the ablation results of this design in Qwen3.5-4B below:
| Configuration | PwAUC | Spearman | Accuracy | QWK |
|---|---|---|---|---|
| Regression head only | 0.8431 | 0.7124 | 0.5644 | 0.6713 |
| Regression + classification heads | 0.9116 | 0.8271 | 0.6701 | 0.7878 |
| Absolute improvement | +0.0685 | +0.1147 | +0.1057 | +0.1165 |
- Therefore, while our paper does not aim to propose a novel backbone, we have conducted well-motivated research on data collection, assessment criteria, and efficient modeling specifically tailored to CHIAA task.
2) Distinction from existing HIAA. First, we would like to clarify that, to the best of our knowledge, after HumanAesExpert proposed HIAA, there has been no further research dedicated to it. Moreover, HumanAesExpert did not open-source its HumanBeauty dataset, whereas we plan to release our dataset upon acceptance. The distinction rests on three aspects:
- Data distribution and quality: The HumanBeauty dataset was collected from existing datasets such as AVA and SCUT-FBP5500, which suffer from limitations such as being single-dimensional and containing limited views (see Figure 1). In contrast, our data is sourced from carefully curated cinematic frames and movie datasets, featuring superior lighting, shadow, and facial expressions. This helps the model learn aesthetics professionally crafted by cinematographers.
- Evaluation criteria and task demands. Previous HIAA works overlooked factors such as atmosphere and emotion, which are more abstract and challenging dimensions in image aesthetic assessment. We incorporate these factors into our evaluation criteria, making the CHIAA task inherently more difficult than HIAA.
- Performance gap: As demonstrated in our experiments, the existing HumanAesExpert-8B achieves only 0.7017 PwAUC on CHIAA-Bench, whereas CineAesNet reaches 0.9211. Furthermore, current MLLMs also yield suboptimal performance. These results highlight an unmet evaluation need and support CHIAA-Bench as a resource for measuring and improving alignment with human cinematic aesthetic judgments.
We sincerely hope the reviewer will reconsider the contribution of the task, benchmark, criteria, and model presented in this paper, and rating in light of these clarifications. We welcome further discussion and appreciate your time.
To Reviewer Gn4t. Based on your considerations, we have presented the innovative points of our paper and its differences from HIAA. If you are willing to re-evaluating your score according to our feedback, we would be very grateful. If you have any other unresolved issues, please let us know. Thank you again.
Rebuttal for Reviewer Gn4t
We thank the Reviewer Gn4t for recognizing the benchmark as the main contribution and for the constructive questions on model design and efficiency.
1) Contribution of design vs. data.
Table 3 isolates the two factors under the same EVA02 backbone: model-score inference reaches 0.8449 while full training reaches 0.9211, indicating the gain mainly comes from CHIAA annotations. For a brand-new task like CHIAA, we believe suitable data is the more important contribution.
2) Training configuration.
As shown in Appendix B.1 and Figure 7, the visual encoder is fully frozen during training; only the LoRA adapters and the two lightweight heads are trained. LoRA is injected into the LLM self-attention modules (
q/k/v/o_proj) with rank=16, alpha=32, dropout=0.05. The number of trainable parameters is 3.15M (4B), 3.93M (9B), and 10.49M (27B), i.e., about 0.04–0.08% of the base model. Other hyperparameters are given in the appendix. At inference, the dual heads are attached to the MLLM and take the hidden state of the last non-pad token in a single forward pass. We will add these details in the revision.3) Efficiency.
The table below compares the inference cost of CineAesNet and fine-tuned small MLLMs. For a deployment-scale projection over 10M cinematic frames: CineAesNet takes 1.1h on 8 GPUs, while the 4B model takes 23h and the 27B model takes 68h on the same 8 GPUs. CineAesNet is far better suited to tens-of-millions-scale production data.