acceptodds
Withdrawn from ICLR 2027

Aligning Understanding and Scoring for Data-Efficient Photographic Aesthetic Assessment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In this work, we study Photography Aesthetics Assessment (PAA), which aims to provide comprehensive aesthetic evaluation of photographs through both qualitative understanding and quantitative scoring. However, reliable photography aesthetic scoring remains challenging because large-scale aesthetic analyses can be obtained relatively easily, while high-quality and consistently calibrated human ratings are costly and difficult to scale. To address these challenges, we propose AesAlign, a data-efficient framework that decouples photography aesthetic understanding from quantitative score calibration. First, we establish a unified photography aesthetic evaluation protocol and construct large-scale fine-grained aesthetic analyses for 500K smartphone photographs. We further collect high-quality human-verified overall and dimension-level ratings for 50K images to provide reliable quantitative supervision. Based on these complementary supervision signals, AesAlign learns photography aesthetic knowledge from scalable analyses and subsequently calibrates the learned representations to human rating scales using a lightweight scoring probe. Extensive experiments demonstrate that AesAlign achieves strong performance under limited human supervision and exhibits robust cross-dataset generalization, enabling comprehensive and interpretable photography aesthetics assessment with reliable overall and multidimensional scores. Our models, training code, and dataset will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Photography Aesthetic Assessment, and Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K7gQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies photography aesthetic assessment (PAA) and proposes AesAlign, a two-stage framework that decouples aesthetic understanding from score calibration. Stage 1 LoRA-tunes Qwen3.5-2B on 450K MLLM-generated aesthetic analyses (produced by Qwen3.7-Plus under a taxonomy of 4 general dimensions + 3 theme-specific dimensions over 10 themes; these texts also embed the generator's own scores). Stage 2 freezes the model, mean-pools layer-22 hidden states over instruction-token positions from a single forward pass, and fits independent Ridge probes per scoring target on 45K expert-calibrated ratings (5K held out for evaluation). The expert ratings are collected with a coarse-to-fine protocol (10-level banding then 1-10 within band, escalation and trimmed averaging) with reference anchors, shared criterion definitions, and blinding to MLLM outputs. Experiments re-annotate the AADB and PARA test sets under this protocol and report SRCC/PLCC against zero-shot and supervised baselines, plus ablations on backbone, score head, pooling, data scale, and stage combinations. The headline claims are +0.073 SRCC over vanilla score fine-tuning under equal expert-rating budgets, and better cross-dataset generalization.

Critical strengths

S1. The expert-calibrated annotation resource and protocol. The 50K fine-grained ratings (45K calibration + 5K evaluation) collected under a coarse-to-fine protocol with reference anchors, criterion definitions, and blinding to MLLM outputs is a genuine asset for PAA research; the protocol details (Appendix A.2) are above the community's current standard for this task. This is, in my assessment, the paper's core contribution. S2. The sample-efficiency finding. Table 3 shows the Ridge-probe calibration reaching 0.739 SRCC with 450 expert-rated images, exceeding Score SFT trained on all 45K (0.706). Quantifying this supervision asymmetry (cheap text vs. expensive calibrated ratings) is a useful empirical message, and the ablation coverage (Tables 4-5) is generally thorough.

Critical weaknesses

W1. The supervised baselines in Table 2 are under-specified, so the central comparison is not interpretable. The paper states baselines are adapted "following each method's training procedure" on AesAlign-Calibrated, but gives no details: input format, score mapping (e.g., how the 1-100 labels map to Q-Align's discrete buckets), LoRA vs. full fine-tuning, hyperparameters, or convergence criteria. Given that InternVL3-8B degrades after adaptation on AADB (0.684 -> 0.670), I cannot exclude that the supervised baselines are under-tuned, which directly undermines the claim that AesAlign outperforms specialized and general MLLM baselines. This is the paper's most important comparison and it is not reproducible from the manuscript. W2. The method contribution reduces to text-bottlenecked distillation, and the decisive baseline is missing. All 450K analyses (and the scores embedded in them) are generated by Qwen3.7-Plus; the student (Qwen3.5-2B) merely absorbs this via LoRA before a linear probe is calibrated on 45K ratings. The paper never compares against directly probing the teacher's (Qwen3.7-Plus) features with the same Ridge heads — a baseline that would plausibly match or exceed the student, since the text bottleneck can only discard information. Table 5's "no Stage 1" row (0.686) probes the un-adapted 2B, which does not address this. Consequently, what remains beyond distillation is thin: the decoupling claim is further weakened because Stage 1 texts embed the generator's scores, suggesting Stage 2 may partly learn a distribution/format correction rather than aesthetic understanding.

Questions for authors

Q1 (clarification, affects W1): Please specify the exact supervised adaptation protocol for each baseline in Table 2: input format, how AesAlign's 1-100 labels map to each method's output (in particular Q-Align's discrete bucketing), LoRA vs. full fine-tuning, learning rates/epochs, and convergence criteria. Also explain why InternVL3-8B degrades after adaptation on AADB. Q2 (clarification, affects W2): Have you tried fitting the same Ridge probes on the frozen features of Qwen3.7-Plus (the analysis generator) with the 45K ratings? If the student outperforms this teacher-probe baseline, please report it; if not, how do you position the method other than as distillation plus protocol alignment? Q3 (clarification): AesAlign-500K, AADB, and PARA are all Flickr-sourced. Was any cross-dataset near-duplicate removal performed (e.g., SSCD/FAISS retrieval between AesAlign-500K and the AADB/PARA test images)? If not, the cross-dataset SRCC gains in Table 2 may partly reflect image overlap between training and "external" test sets. Q4 (clarification, affects W2): Since the Stage 1 texts embed Qwen3.7-Plus's own scores, can you disentangle how much of the Stage 1 benefit comes from aesthetic understanding versus distribution/format correction — e.g., by re-running Stage 1 on texts with all numerical scores stripped?

Reviewer_Qi3gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper consists of two components: the construction of photography aesthetics assessment data and the design of the model architecture. The task targets grounded aesthetic analyses, reliable overall aesthetic scores, and dimension-wise subscores.

First, building the AesAlign-500K dataset. It comprises 500K smartphone photographs paired with fine-grained aesthetic analyses generated by MLLMs, among which 50K samples additionally carry expert-calibrated overall and dimension-level ratings.

Second, proposing AesAlign, a two-stage framework. In the first stage, it learns photographic aesthetic representations from the large-scale aesthetic analysis corpus. In the second stage, the regression-based scoring heads are calibrated using a smaller expert-annotated set.

Critical strengths

A relatively complete dataset and evaluation protocol for photographic aesthetics. The attribute design and coarse-to-fine scoring process provide a richer benchmark for PAA. This is valuable because photographic aesthetics assessment requires dimension-level explanations and sub-scores, rather than only a scalar ranking result.

Critical weaknesses

Limited validation of stage1. Stage 1 depends on MLLM-generated analyses, but the paper mainly validates downstream scores. It does not sufficiently show whether the learned analyses are more accurate, more faithful, or less biased than the teacher outputs. Since the method claims aligned understanding and scoring, independent analysis-quality evaluation is important.

Questions for authors

1.How was the quality of the generated aesthetic analyses evaluated independently from score prediction? Do human experts prefer AesAlign analyses over the teacher MLLM outputs or other baselines? 2.For the AADB/PARA evaluation, did you conduct new human re-annotations using the same expert calibration protocol as AesAlign-Eval? If so, this seems equivalent to applying the AesAlign annotation pipeline to images from existing open-source benchmarks. In that case, could the youclarify in what sense the AADB/PARA evaluation provides an independent external benchmark, rather than another AesAlign-style evaluation set differing mainly in image source? 3.I notice the scarcity of low-scoring samples in the data distribution. Did you verify if the model has a distribution bias and tends to give undesirably high scores to clearly low-quality images?

Reviewer_K8xbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AesAlign, a framework for photographic aesthetics assessment that produces both textual analyses and numerical scores. It also presents AesAlign-500K, a dataset of 500,000 smartphone photographs with MLLM-generated analyses and a 50,000-image subset with expert-calibrated ratings. AesAlign first learns from the analyses, then fits linear scoring probes on frozen model representations using expert ratings. The reported results show gains over direct score fine-tuning, including on AADB and PARA test images that were re-annotated using the paper’s rating protocol.

Critical strengths

The two-stage approach is easy to follow, and the analysis-supervision and score-calibration ablations help motivate it. The manuscript describes the dataset construction and hierarchical rating procedure in useful detail.

Critical weaknesses
  1. The paper describes the expert-rating procedure but does not report inter-rater agreement or uncertainty in the final ratings. It also relies on MLLM-generated analyses at scale without a clear human audit of their accuracy.
  2. AADB and PARA test images are re-annotated using AesAlign’s rating protocol. This evaluates transfer under a shared protocol, but does not directly show performance against the original benchmark ratings.
  3. Since AesAlign is built from Flickr photographs, please report checks for overlap or near-duplicates between its training data and the AADB/PARA test images. Also describe whether the random split controls for photographer or other sources of similarity that could affect evaluation.
  4. Explain how images classified as “Other” are handled, since the prompt assigns them no theme-specific dimensions while the method describes eight outputs. For a planned public release, document the image-source terms, access conditions, and relevant privacy safeguards.
  5. Correct small wording and formatting issues, including “imagess,” “languagebased,” inconsistent hyphenation, and the missing “ratings” in the third contribution.
Questions for authors

What inter-rater agreement did the expert ratings achieve, and what human audit was conducted on the MLLM-generated analyses? This evidence would affect my assessment of supervision quality. How do results on AADB and PARA compare when evaluated against their original ratings as well as the paper’s re-annotated ratings, after checking for overlap with AesAlign’s training data? This would help establish how broadly the cross-dataset findings generalize.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.