acceptodds
Under review as a conference paper at ICLR 2027

CHIAA-Bench: Aligning Cinematic Human Image Aesthetic Assessment with Human Aesthetic Judgments

Abstract

Image aesthetic assessment has been widely studied for generic images, but remains underexplored for cinematic human images, whose aesthetic quality is mainly determined by human-centered cues such as facial clarity, expression, and atmosphere. To address this gap, we introduce a new task, **Cinematic Human Image Aesthetic Assessment (CHIAA)**, which aims to evaluate the aesthetic quality of cinematic human images from movies and TV series. We further present **CHIAA-Bench**, a new benchmark constructed from movie and TV frames as well as open-source cinematic datasets through human image filtering and a two-stage annotation pipeline with model scoring and expert refinement. Our annotation protocol covers four aesthetic dimensions and ten fine-grained sub-dimensions, producing interpretable five-level labels for cinematic human image aesthetics. Based on this benchmark, we develop **CineAesNet**, a lightweight model for efficient cinematic aesthetic assessment. Experiments show that CineAesNet consistently outperforms existing open-source aesthetic models and strong multimodal large language models on CHIAA-Bench while using substantially fewer parameters. In addition, our analysis shows that model-generated scores are useful for candidate filtering and low-resource warmup, whereas expert supervision remains essential for reliable learning of cinematic human aesthetics.

Primary area:
datasets and benchmarks
Keywords:
Cinematic aesthetics, Image aesthetic assessment, Benchmark, Multimodal models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Gn4tOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CHIAA-Bench, containing 20.8k expert-annotated images, with an assessment protocol covering four major dimensions and ten fine-grained attributes. It further proposes CineAesNet, a lightweight EVA02-based aesthetic predictor. Experiments show strong performance against existing aesthetic models and MLLMs on both in-domain and cross-source evaluations. The authors also investigate model-assisted annotation and the adaptation of MLLMs to this task.

Critical strengths
  1. Existing aesthetic benchmarks do not adequately capture the characteristics of cinematic human images. CHIAA-Bench addresses this gap with a well-motivated evaluation protocol and expert-refined annotations. I consider this the main contribution of the paper.

  2. The authors compare against various aesthetic models and MLLMs, evaluate cross-source generalization, and investigate the contributions of model-generated scores and human annotations. These experiments provide useful evidence for the value of the proposed dataset.

Critical weaknesses
  1. CineAesNet essentially combines a pretrained EVA02 backbone with a lightweight prediction head and conventional training objectives. It is unclear how much of the improvement comes from the model design rather than the new training data. The architectural contribution appears relatively limited.

  2. The authors demonstrate that fine-tuned 4B and 9B MLLMs already achieve performance close to CineAesNet. However, the fine-tuning configuration, particularly whether the visual encoder is updated, is not sufficiently clear. More controlled comparisons with smaller MLLMs and different visual adaptation strategies would help establish the advantage of CineAesNet.

Questions for authors
  1. When fine-tuning the 4B, 9B, and 27B MLLMs, is the visual encoder frozen? Which modules are updated through LoRA, and what are the corresponding training configurations?

  2. Can the authors provide inference latency, throughput, and memory comparisons between CineAesNet and the fine-tuned smaller MLLMs under comparable settings? This would better substantiate the claimed efficiency advantage.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Gn4t

We thank the Reviewer Gn4t for recognizing the benchmark as the main contribution and for the constructive questions on model design and efficiency.

1) Contribution of design vs. data.

Table 3 isolates the two factors under the same EVA02 backbone: model-score inference reaches 0.8449 while full training reaches 0.9211, indicating the gain mainly comes from CHIAA annotations. For a brand-new task like CHIAA, we believe suitable data is the more important contribution.

2) Training configuration.

As shown in Appendix B.1 and Figure 7, the visual encoder is fully frozen during training; only the LoRA adapters and the two lightweight heads are trained. LoRA is injected into the LLM self-attention modules (q/k/v/o_proj) with rank=16, alpha=32, dropout=0.05. The number of trainable parameters is 3.15M (4B), 3.93M (9B), and 10.49M (27B), i.e., about 0.04–0.08% of the base model. Other hyperparameters are given in the appendix. At inference, the dual heads are attached to the MLLM and take the hidden state of the last non-pad token in a single forward pass. We will add these details in the revision.

3) Efficiency.

The table below compares the inference cost of CineAesNet and fine-tuned small MLLMs. For a deployment-scale projection over 10M cinematic frames: CineAesNet takes 1.1h on 8 GPUs, while the 4B model takes 23h and the 27B model takes 68h on the same 8 GPUs. CineAesNet is far better suited to tens-of-millions-scale production data.

Model#ParamsThroughput (img/s, e2e)Peak memorySpeed ratioMemory ratio
CineAesNet (Light)86.4M324.40.86 GB1×1×
CineAesNet-4B4,543M15.112.5 GB21.5×14.5×
CineAesNet-9B9,415M13.122.6 GB24.7×26.2×
CineAesNet-27B27,369M5.161.1 GB63.6×70.9×
Reviewer_Gn4tOpenReview ↗

Post rebuttal

I appreciate the authors' detailed rebuttal. The clarified MLLM training configurations and additional efficiency comparisons adequately address my original questions. The supplementary analyses also strengthen the experimental evidence.

However, I agree with Reviewers Qvt6 and cie3 regarding the limited technical novelty. CineAesNet relies on a standard backbone and conventional training objectives, without meaningful designs tailored to cinematic human aesthetics. Moreover, the distinction between CHIAA and existing human image aesthetic assessment remains insufficiently justified.

While I recognize the value of the benchmark and its annotations, these fundamental concerns remain unresolved. I therefore lower my rating from to 2.

AuthorsOpenReview ↗

Re-response to Reviewer Gn4t

We thank the reviewer for confirming that our responses addressed the original questions. We would like to clarify the two remaining concerns.

1) Technical novelty.

  • On the one hand, the novelty of this work lies in task formulation, benchmark construction, and annotation protocol design. To the best of our knowledge, research in the field of human aesthetic assessment remains relatively scarce, and studies specifically targeting the cinematic domain are virtually nonexistent. We believe that contributing a benchmark based on lightweight, task-specific models is a well-established research paradigm. For this task, we designed a ten-dimensional assessment protocol, automated human-image filtering (Appendix A.1), and explicit expert grading criteria to construct a usable cinematic human aesthetic benchmark.

  • On the other hand, we have not neglected the design for cinematic human image aesthetic assessment. Our task-specific designs also include score fusion across MLLMs, aesthetic models, and quantitative visual cues (Appendices A.3–A.4). For MLLM adaptation, we also adopt a well-justified architecture, we added an extra Regression Head and a Classification Head specifically tailored to this task. We additionally report the ablation results of this design in Qwen3.5-4B below:

ConfigurationPwAUCSpearmanAccuracyQWK
Regression head only0.84310.71240.56440.6713
Regression + classification heads0.91160.82710.67010.7878
Absolute improvement+0.0685+0.1147+0.1057+0.1165
  • Therefore, while our paper does not aim to propose a novel backbone, we have conducted well-motivated research on data collection, assessment criteria, and efficient modeling specifically tailored to CHIAA task.

2) Distinction from existing HIAA. First, we would like to clarify that, to the best of our knowledge, after HumanAesExpert proposed HIAA, there has been no further research dedicated to it. Moreover, HumanAesExpert did not open-source its HumanBeauty dataset, whereas we plan to release our dataset upon acceptance. The distinction rests on three aspects:

  • Data distribution and quality: The HumanBeauty dataset was collected from existing datasets such as AVA and SCUT-FBP5500, which suffer from limitations such as being single-dimensional and containing limited views (see Figure 1). In contrast, our data is sourced from carefully curated cinematic frames and movie datasets, featuring superior lighting, shadow, and facial expressions. This helps the model learn aesthetics professionally crafted by cinematographers.
  • Evaluation criteria and task demands. Previous HIAA works overlooked factors such as atmosphere and emotion, which are more abstract and challenging dimensions in image aesthetic assessment. We incorporate these factors into our evaluation criteria, making the CHIAA task inherently more difficult than HIAA.
  • Performance gap: As demonstrated in our experiments, the existing HumanAesExpert-8B achieves only 0.7017 PwAUC on CHIAA-Bench, whereas CineAesNet reaches 0.9211. Furthermore, current MLLMs also yield suboptimal performance. These results highlight an unmet evaluation need and support CHIAA-Bench as a resource for measuring and improving alignment with human cinematic aesthetic judgments.

We sincerely hope the reviewer will reconsider the contribution of the task, benchmark, criteria, and model presented in this paper, and rating in light of these clarifications. We welcome further discussion and appreciate your time.

To Reviewer Gn4t. Based on your considerations, we have presented the innovative points of our paper and its differences from HIAA. If you are willing to re-evaluating your score according to our feedback, we would be very grateful. If you have any other unresolved issues, please let us know. Thank you again.

Reviewer_cie3OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors present a relatively new task of cinematic human image aesthetic assessment. A new dataset CHIAA-Bench was built and annotated by combining model scoring and expert refinement. A new task-specific model called CineAesNet was also proposed based on CHIAA-Bench. The authors have done extensive experiments and comparisons to demonstrate the advantage of the proposed method. This work follows the standard practice of papers in the era of large models. The paper addresses a very specific subtask of image aesthetic assessment. It is generally well-written and easy to follow.

Critical strengths
  1. Cinematic human image aesthetic assessment is a relatively new task in the IAA community.
  2. The authors have conducted a systematic study of the task, including problem formulation, dataset construction with annotation, model design.
  3. Experimental results are comprehensive with the latest models in this area.
Critical weaknesses
  1. The motivation of cinematic human image aesthetic assessment is not strong, considering its similarity to human image aesthetics.
  2. The technical contributions are incremental in general lacking innovative designs.
  3. Cinematic human aesthetics is too specific, and this also means that the group of people interested in it is very small. What are the key differences between human image aesthetics and cinematic human image aesthetics?
  4. What is the design logic of model scoring and expert refinement in dataset annotation? It is not clear what is the role of model scoring.
  5. Human aesthetics is closely related to the beauty of human face. How to decouple the factors of facial aesthetics in the process of data annotation and model construction?
  6. Are atmosphere and decoration really factors of cinematic human aesthetics instead of image aesthetics? Particularly the atmosphere, it is also a key factor in general image aesthetics.
Questions for authors
  1. Authors are advised to include a more in-depth analysis of the motivation of the CHIAA task. Why is it important and what kinds of applications does it have?
  2. What are the key differences between human image aesthetics and cinematic human image aesthetics? This is important and should be discussed in combination with the model design. In this way, the technical contributions of the proposed method can be strengthened.
2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer cie3

We thank the Reviewer cie3 for recognizing the relative novelty of the task and the systematic study, and for the questions that helped us clarify the motivation and the distinction from prior work.

1) Motivation and applications.

As stated in the Introduction (Lines 48–80): on one hand, CHIAA places much stronger demands on emotion and atmosphere, which we control through the four aesthetic dimensions during annotation, making it stricter than generic HIAA; on the other hand, prior HIAA work (only HumanAesExpert) did not release its human aesthetic dataset, and its model performs poorly (Figures 4 and 8), while recent HIAA work offers neither better models nor better data. This demonstrates the necessity and value of our task.

In practice, cinematic human image aesthetic assessment helps us rapidly select suitable promotional frames from tens of millions of movie/TV series candidate images, with 21.5–63.6× faster than MLLMs (see the rebuttal for Reviewer Gn4t-3). Moreover, human evaluation confirms that our model meets the demands of real applications including promotion, further poster editing, delivery, and recommendation, consistently outperforming existing open-source MLLMs and aesthetic models.

2) Novelty.

The core novelty of this work lies in the perspective and standards of data selection. Our goal is to train a high-performance and high-efficiency end-to-end model with suitable data. As shown in Table 6, there is currently no suitable, open-source dataset dedicated to human-centric aesthetic scoring in the cinematic domain. To build such a dataset, we developed an automated data selection pipeline (Appendix A.1) that filters roughly the top 1% of suitable samples from millions of cinematic frames, drastically reducing the manual effort required compared with direct human annotation.

3) Key differences from HIAA.

  1. Data distribution: cinematic frames involve professional cinematography and lighting, differing substantially from web-collected photos, and transfer better to downstream cinematic applications.
  2. Evaluation criteria: CHIAA incorporates higher-level criteria such as atmosphere and emotion, aiming to find outstanding cinematic frames, whereas HIAA focuses more on person-level attributes and neglects atmospheric and decorative aesthetics.
  3. Task difficulty: CHIAA requires deeper understanding such as atmosphere and emotion, whereas HIAA is a lower-difficulty task. CHIAA evaluation covers the person-level factors of HIAA, whereas HIAA models cannot be directly applied to CHIAA.
  4. Empirical gap: HumanAesExpert-8B achieves only 0.7017 PwAUC on CHIAA-Bench while CineAesNet reaches 0.9211, showing that existing HIAA models cannot meet the needs of cinematic human image assessment.

4) Model scoring.

Our experiments in Sec. 4.4 demonstrate the role of model scores:

  1. pure model-score inference achieves PwAUC 0.8449, already surpassing most existing aesthetic models, showing the task can be bootstrapped at low cost;
  2. in the low-resource setting, pretraining on model scores followed by refinement with only 2k human labels substantially outperforms training on 2k labels alone (Table 3);
  3. annotation assistance: model scores provide additional verification during annotation, through which we corrected 382 samples. The table below shows performance on the test set before and after this correction, indicating that model scores effectively assist human annotation by surfacing the most error-prone boundary samples for expert review at low cost. We will clarify this in the revision.
ScorerPwAUC (before → after)QWK (before → after)Spearman (before → after)
CineAesNet0.8994 → 0.9211 (+0.0217)0.7767 → 0.8130 (+0.0363)0.8055 → 0.8415
Qwen3.8-27B 10-dim Inference0.7858 → 0.80330.5668 → 0.60190.5984 → 0.6340

5) Facial aesthetics.

Model scoring includes attractiveness and expression as sub-dimensions, with detailed weights in Appendix A.4. Note that our goal is an overall aesthetic score rather than evaluation of any single sub-dimension. Table 10 in the appendix shows the model's capability on the attractiveness dimension, indicating that the final prediction is a comprehensive combination of multiple dimensions rather than dominated by facial attractiveness.

6) Atmosphere.

Atmosphere is indeed also a factor in generic IAA. In our protocol it is one evaluation dimension among ten and not the dominant factor. The poor performance of generic IAA models on CHIAA-Bench shows they cannot be directly applied to human-centric cinematic assessment. Overall, CHIAA comprehensively evaluates all dimensions, which yields both better final performance and a more efficient model.

To Reviewer cie3. The discussion period is only one day left. In our response, we provided further analysis of motivation and applications, elaborating more on the differences between us and the HIAA task. If you consider re-evaluating your score based on our response, we would be very grateful. If you have any other unresolved issues, please let us know. Thank you again.

Reviewer_Qvt6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CHIAA, a new task for cinematic human image aesthetic assessment, together with CHIAA-Bench and CineAesNet (a lightweight EVA02-based predictor). Overall, the work addresses a reasonable application gap and reports strong empirical gains over MLLMs and existing aesthetic models, but the methodological novelty of CineAesNet is thin and several design choices, such as weight selection, expert reliability, label mapping, are insufficiently justified.

Critical strengths

The task framing and benchmark construction are solid contributions. The ten-dimension protocol, two-stage model-plus-expert annotation, shot-level deduplication, and inclusion of both in-domain and out-of-domain splits provide a reusable resource.

Critical weaknesses

Methodological novelty of CineAesNet is limited. It is essentially an EVA02 backbone with a standard distribution-prediction head trained with a common EMD + PLCC + pairwise-rank loss combination, with no architectural or algorithmic component designed specifically for cinematic or human-centric aesthetics, so the paper's technical contribution reduces almost entirely to the dataset.

Questions for authors

1 What is the inter-annotator agreement among the six cinematic experts (e.g., Cohen's kappa or QWK on the double-annotated subset), and how many images actually required third-expert adjudication?

2 Since the training set is heavily imbalanced (only 94 "excellent" and 1,510 "good" vs. 5,573 "normal" plus 5,000 injected "bad" samples in Table 5), how does CineAesNet perform specifically on the "excellent" and "good" categories, and could the strong QWK partly reflect this skew rather than fine discrimination at the top end?

3 Why does the non-uniform label mapping {0, 0.25, 0.5, 0.75, 1} differ from the stated motivation of "non-uniform spacing that matches human perception," given that the chosen mapping is actually uniform in 0.25 steps?

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Qvt6

We thank the Reviewer Qvt6 for the positive assessment of our task framing, benchmark construction, and experiments, and for the constructive questions.

1) Methodological novelty.

As stated in our main contributions, the core contributions of this work are the task formulation (CHIAA), the dataset (CHIAA-Bench), and the annotation protocol. Our goal is to train a high-performance, high-efficiency end-to-end model with suitable data. CineAesNet, with only 86.4M parameters, is 21.5–63.6× faster than fine-tuned MLLMs (see rebuttal for Reviewer Gn4t-3), a deliberate design trade-off for deployment over tens of millions of cinematic frames. Our designed for cinematic and human-centric aesthetics are mainly embodied in the annotation standards and the fine-grained scoring protocol; in addition, Appendix A.3 provides quantitative visual cues for dimensions such as face sharpness and expression.

2) Inter-annotator agreement.

Thank you for pointing this out. We report the average agreement statistics of the annotation process:

RawCohen κLinear κQWKFleiss κ
0.6040.4810.6630.8110.570

Among the 20.8k samples, 6,987 images required third-expert adjudication, and 382 samples were further corrected in the re-check stage. We will add these details in the revision.

3) Class imbalance.

The imbalance reflects the genuine scarcity of excellent samples in real data.

  1. Among the reported metrics, EG/R (excellent+good vs. the rest) is 0.9273 and EG/N (excellent+good vs. normal) is 0.8400, showing the model retains discriminative power on sparse high-end classes.
  2. We further provide E/R (excellent vs. the rest) and E/G (excellent vs. good) metrics below, which show the model can distinguish excellent from other categories; since excellent samples are limited, these evaluations are less stable.
  3. Methodologically, QWK already corrects for chance agreement: under the current marginal distribution, a model that only predicts the majority class normal (37.3%) has an accuracy ceiling of 37.3% and a near-zero QWK. Hence a high QWK cannot be produced by skew alone and is a strong result under imbalance.
ModelE/R (video)E/G (video)E/R (ShotBench)E/G (ShotBench)E/R (MovieChat)E/G (MovieChat)
HumanAesExpert0.75480.59710.46450.32070.74820.5476
FGAesQ0.86340.66780.90050.82140.85570.7323
OneAlign0.87340.68560.93250.86520.86140.6434
HumanVLM0.63740.56730.50760.48020.62830.5913
Qwen3.7-Plus0.86160.59900.72310.47160.92660.7884
Qwen3.8-Max0.91650.70540.85560.69920.93380.7955
Kimi-K30.85370.65890.84770.64920.92300.7158
CineAesNet0.91760.68830.88730.80500.84660.6386

4) Label mapping.

Thank you for pointing this out; it is a typo in our writing. The "non-uniform" design in the paper refers to the fusion weights of the model-score dimensions (Appendix A.4, Details of Model-Score Fusion), not to the mapping of the five-level labels. We will fix the wording in the revision. In addition, we run an ablation on label mappings under the same configuration, showing that this setting does not affect the conclusions of the paper:

Label mappingPwAUCQWKAccLabel Spearman
uniform ({0, .25, .5, .75, 1})0.92110.81300.69920.8415
nonuniform-1 ({0, .20, .45, .72, 1})0.91360.80820.68740.8352
nonuniform-2 ({0, .30, .55, .78, 1})0.91530.81170.69560.8376

To Reviewer Qvt6. Thank you for your reading. The discussion period is only one day left. We have answered your questions regarding innovation and other issues. You can also refer to the detailed response Re-response to Reviewer Gn4t for more information. If you consider re-evaluating your score based on our response, we would be very grateful. If you have any other unresolved issues, please let us know. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.