acceptodds
Under review as a conference paper at ICLR 2027

Learning Aesthetic Preferences in Contextually Related Image Collections

Abstract

The rapid advancement of smart devices and social media has significantly increased the user demand for aesthetic applications such as album management, photo sharing, and recommendation. In these real-world scenarios, users generally select preferred photographs from a collection of images depicting the same scene, event, or subject, making aesthetic preference inherently context-dependent. However, existing aesthetic evaluation methods typically focus on either context-independently or highly similar image series, limiting their ability to capture preferences within contextually related image collections. To address this gap, we introduce ContextAes-40K, a novel dataset comprising 39,033 images organized into 8,069 contextually related collections, with pairwise aesthetic preference annotations within each collection. To further evaluate performance in real-world user-curated collections, we collect ContextAes-Album, an independent real-user album dataset with user-provided aesthetic rankings. Building upon ContextAes-40K, we propose ContextAes, an MLLM-based aesthetic model trained through a three-stage training strategy. Extensive experimental results show that ContextAes achieves state-of-the-art performance on ContextAes-40K, exhibits strong generalization ability on other aesthetic benchmarks, and also demonstrates superior zero-shot performance on ContextAes-Album, highlighting its ability to identify aesthetic preferences within contextually related image collections.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image aesthetics assessment, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Qd9FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that aesthetic preference should be modeled within contextually related image collections (images of the same scene, event or subject, or edited variants of one source image) rather than for isolated images. Its contributions are: (1) ContextAes-40K, 39,033 images organized into 8,069 collections, drawn from RedNote, ETH3D and HPatches (natural photography) and Lightroom (editing); (2) ContextAes-Album, 494 user-curated albums with owner-provided top-5 rankings, used for zero-shot evaluation; and (3) ContextAes, a Qwen3.5-9B model trained in three stages (SkillOpt skill optimization, LoRA SFT, GRPO). ContextAes achieves state-of-the-art results on ContextAes-40K, generalizes strongly across other aesthetic benchmarks, and shows excellent zero-shot performance on ContextAes-Album, demonstrating its ability to capture aesthetic preference within contextually related image collections. The paper is clearly written and the dataset construction is documented in detail.

Critical strengths
  1. The authors purpose modeling aesthetic preference within contextually related image collections rather than over isolated images.
  2. They construct ContextAes-40K and ContextAes-Album, two datasets that cover natural photography and image editing and support both training and zero-shot evaluation.
  3. They propose ContextAes, a Qwen3.5-9B model trained in three stages, and extensive experiments show state-of-the-art accuracy on ContextAes-40K, strong generalization to other aesthetic benchmarks, and excellent zero-shot performance on ContextAes-Album.
Critical weaknesses
  1. ContextAes-40K is not shown to be an effective training resource beyond the authors' own pipeline, and no non-MLLM model is evaluated on it. Because the dataset is used only to train ContextAes, it remains unclear whether it can support aesthetic preference learning within contextually related image collections in other models.
  2. Latency and Cost of MLLM-Based Inference ContextAes is built on a 9B parameter MLLM (Qwen3.5-9B) with a three-stage pipeline . In real-time album management, running a 9B model on a smartphone or even a server for thousands of images is computationally expensive compared to traditional aesthetic scorers. The paper does not provide latency benchmarks or on-device deployment tests. While it claims "superior zero-shot performance," it does not address the energy/compute trade-off required for daily use.
Questions for authors

1.What capability or decision does this setting enable that the closely-similar-series setting (FGAesthetics) does not? 2.What is the breakdown of ContextAes-40K by source (images, collections, retained pairs), and how does pairwise accuracy differ across sources, in particular RedNote vs. ETH3D/HPatches?

7 comments

To reviewer Qd9F: Thanks for the constructive comments. Responses to specific comments are given below, which have also been incorporated into the revised paper.

  1. W1: Generalizability of ContextAes-40K beyond the proposed model. Thanks for raising this important concern. We agree that the effectiveness of ContextAes-40K should be demonstrated beyond our proposed training pipeline. To this end, we have fine-tuned several models with different architectures on the same ContextAes-40K training split, including conventional visual models (ResNet-50, ViT-B/16, and CLIP-RN50) and aesthetic-specialized models (ArtiMuse and Q-Align). All models have been evaluated on the same held-out test set. The results are presented below:

    ModelNatural AccNatural F1Editing AccEditing F1Overall AccOverall F1
    ResNet-5062.78%62.74%71.08%71.05%66.56%66.53%
    ViT-B/1661.74%61.64%70.20%70.17%65.60%65.54%
    CLIP-RN5065.41%65.36%72.93%72.91%68.84%68.80%
    ArtiMuse (Original)56.38%56.31%53.10%53.10%54.89%54.87%
    ArtiMuse67.29%67.25%75.92%75.89%71.22%71.19%
    Q-Align (Original)61.39%61.26%59.41%59.41%60.49%60.45%
    Q-Align67.97%67.93%77.32%77.30%72.23%72.20%

    These results demonstrate that ContextAes-40K supports aesthetic preference learning across different model architectures, rather than being limited to our proposed MLLM-based training pipeline. In particular, conventional visual models can also learn meaningful aesthetic preference representations from the provided training data, while aesthetic-specialized models achieve even stronger performance after fine-tuning. This provides additional evidence of the applicability and utility of ContextAes-40K as a training resource for aesthetic preference evaluation in contextually related image collections.

    The corresponding revisions have been incorporated into Table 1 (Page 7) and Appendix E.1.1 (Page 22, Line 1184--1229), with the modified text highlighted in blue.

  1. W2: Inference efficiency and practical deployment of ContextAes. Thanks for raising this important concern. We acknowledge that directly deploying a 9B-parameter MLLM may introduce considerable computational overhead, particularly for resource-constrained mobile devices. However, our approach is not limited to the 9B model. To explore a more practical deployment solution, we further trained a lightweight 2B-parameter model using the same training methodology and evaluated its performance against larger models. The results are presented below. Both ArtiMuse and Q-Align have been fine-tuned on the same ContextAes-40K training split for a fair comparison.

    ModelNatural AccNatural F1Editing AccEditing F1Overall AccOverall F1
    ArtiMuse67.29%67.25%75.92%75.89%71.22%71.19%
    Q-Align67.97%67.93%77.32%77.30%72.23%72.20%
    ContextAes (2B)71.02%70.77%75.92%75.92%73.25%73.19%
    ContextAes (9B)72.44%72.44%78.71%78.60%75.30%75.27%

    As shown above, the 2B model achieves an overall accuracy of 73.25%, outperforming both fine-tuned ArtiMuse and Q-Align, while maintaining competitive performance compared with the 9B model. Although the smaller model exhibits some performance degradation relative to the 9B version, it provides a more lightweight alternative with a favorable balance between model size and aesthetic preference prediction performance.

    To further evaluate its practical deployment feasibility, we have deployed the 2B model on a vivo X500 Pro smartphone and performed on-device inference using its NPU. The model was quantized to W4A16 (4-bit weights and 16-bit computation) and deployed using MediaTek's MTK LLM SDK, with Neuron 10.0.0 serving as the compiler. The measured inference efficiency is summarized below:

    MetricResult
    Average LatencyApproximately 0.8 s/sample
    Inference PowerApproximately 2.7 W
    Estimated Energy per SampleApproximately 2.16 J

    The measured latency is approximately 0.8 seconds per sample, with an inference power consumption of approximately 2.7 W, corresponding to an estimated energy consumption of 2.16 J per sample.

    These results demonstrate that the proposed training methodology can also be applied to smaller MLLMs, enabling a more practical deployment alternative without requiring the 9B model in resource-constrained environments. While the 9B model provides stronger aesthetic preference prediction performance, the 2B variant achieves competitive accuracy and can run directly on a smartphone NPU with low inference latency and power consumption. This provides a feasible trade-off between prediction performance and computational cost for real-world album management applications.

    Regarding server-side processing of thousands of images, it is relatively uncommon for a single contextually related collection to contain thousands of images, as such collections are typically associated with specific events, subjects, or contexts. Although a user's entire photo library may contain thousands of images, these images generally span multiple events and contexts and can therefore be organized into smaller contextually related collections before aesthetic preference evaluation. In such cases, the pairwise ContextAes model can be applied to each collection separately.

    Nevertheless, for scenarios where an exceptionally large collection needs to be ranked as a whole, we have further developed a score-based variant, ContextAes (Score), to provide a more computationally efficient alternative. Specifically, we attach an MLP-based scoring head to the Qwen3.5-9B backbone and train the model using a pairwise logistic loss. Unlike the pairwise model, which requires autoregressive generation to predict preferences for individual image pairs, the score-based model directly outputs a scalar aesthetic score for each image through the MLP-based head, enabling the ranking of the entire collection based on these scores.

    The performance comparison is presented below:

    ModelNatural AccNatural F1Editing AccEditing F1Overall AccOverall F1
    ArtiMuse67.29%67.25%75.92%75.89%71.22%71.19%
    Q-Align67.97%67.93%77.32%77.30%72.23%72.20%
    ContextAes (Score)70.35%70.30%78.13%78.12%73.87%73.85%
    ContextAes (Pair)72.44%72.44%78.71%78.60%75.30%75.27%

As shown above, ContextAes (Score) achieves an overall accuracy of 73.87%, outperforming fine-tuned ArtiMuse and Q-Align by 2.65 and 1.64 percentage points, respectively. Although its performance is slightly lower than that of ContextAes (Pair), it provides a more computationally efficient alternative for large-scale image ranking.

To further compare their server-side inference efficiency, we have evaluated ContextAes (Score), ArtiMuse, and Q-Align on a single NVIDIA H20 GPU using the same number of test images and the same maximum input resolution. We tested batch sizes of 1, 4, 8, 16, 32, and 64, with 5 repetitions after 5 warm-up batches. We report the end-to-end (E2E) latency and throughput, together with peak allocated and reserved GPU memory. The results are presented in the following table.

Batch SizeModelE2E Latency (ms/img)E2E Throughput (img/s)Peak Allocated (GiB)Peak Reserved (GiB)
1ContextAes (Score)132.207.5615.7316.02
ArtiMuse134.267.4515.4015.55
Q-Align147.296.7915.4315.46
4ContextAes (Score)74.8313.3615.8416.26
ArtiMuse93.1710.7315.6315.97
Q-Align80.6112.4115.6715.71
8ContextAes (Score)61.3316.3116.0016.53
ArtiMuse81.0512.3415.9316.52
Q-Align67.1814.8815.9916.05
16ContextAes (Score)57.1917.4916.3216.79
ArtiMuse76.8313.0216.5317.58
Q-Align57.7317.3216.6216.71
32ContextAes (Score)55.1918.1216.9617.95
ArtiMuse74.6313.4017.7419.69
Q-Align52.1319.1817.8918.04
64ContextAes (Score)48.9020.4518.2420.12
ArtiMuse71.7613.9420.1523.92
Q-Align47.8920.8820.4320.71

As shown above, increasing the batch size improves the inference throughput of all three models while reducing the per-image latency. ContextAes (Score) achieves 48.90 ms/image and 20.45 images/s at a batch size of 64, substantially outperforming ArtiMuse in inference speed and achieving throughput comparable to Q-Align. Notably, ContextAes (Score) maintains a relatively low memory footprint as the batch size increases, requiring only 18.24 GiB of peak allocated memory at batch size 64, compared with 20.43 GiB for Q-Align. These results further support its suitability for server-side ranking of exceptionally large image collections.

Overall, these model variants provide complementary deployment options for different application scenarios. The lightweight 2B pairwise model enables on-device inference on smartphones for typical-sized collections, while the 9B pairwise model provides higher prediction accuracy. For exceptionally large collections, ContextAes (Score) offers a more computationally efficient alternative through direct score prediction. Together, these variants provide flexible trade-offs between prediction performance and computational cost for practical album management applications.

The corresponding revisions have been incorporated into Table 15 (Page 31), Table 16 and 17 (Page 32) and Appendix F.2 and F.3 (Page 33, Line 1748--1787), with the modified text highlighted in blue.

Reviewer_QsfZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark of contextually related natural and edited image collections with pairwise human aesthetic preferences. It additionally constructs an independent album benchmark containing rankings supplied by image-collection providers. The proposed multimodal model combines SkillOpt, supervised fine-tuning, and GRPO-based preference optimization. Evaluation covers pairwise preference prediction, established aesthetic-scoring benchmarks, training-stage ablations, and zero-shot transfer to the independently collected album rankings.

Critical strengths

The dataset addresses a useful setting between independent-image assessment and comparison of near-duplicate images. Contextually related collections are relevant to photograph selection and album organization, and the pairwise human annotations provide a potentially valuable resource for studying relative aesthetic judgment.

The independent album evaluation is a meaningful addition because it tests transfer to preferences supplied by actual collection providers rather than only reproducing the authors’ annotation procedure. The backbone-controlled comparison and training-stage ablations also provide useful initial evidence about the effect of task-specific adaptation.

Critical weaknesses

The evaluation protocol leaves important risks unresolved. The training and test split is not identified as collection- or source-disjoint, and the origin of the validation data used during SkillOpt is unclear. A pair-level split could expose the model to the same images and collection-specific preferences across training and testing. In addition, filtering a fixed fraction of low-confidence or ambiguous comparisons without reporting inter-annotator agreement or unfiltered results likely makes the benchmark systematically easier and limits interpretation of the reported accuracy.

The main comparison lacks training-matched baselines. None is reported as fine-tuned on the same benchmark training split, so the results primarily establish that task-specific adaptation improves over off-the-shelf models rather than that the proposed recipe is superior. This is particularly important because the model still receives only an image pair rather than the full collection, and the training-stage gains are small or inconsistent in several ablations, with no variability estimates reported.

Questions for authors

Is the dataset split performed at the pair, image, collection, or source level, and are the SkillOpt training and validation data strictly disjoint from the test collections?

Can you report inter-annotator agreement, raw majority-vote results, and performance without confidence-based filtering so that label reliability and benchmark difficulty can be assessed?

Can you fine-tune strong competing methods on identical splits and add a full-collection input control to separate the benefits of contextual supervision from ordinary task-specific fine-tuning?

4 comments

To reviewer QsfZ: Thanks for the constructive comments. Responses to specific comments are given below, which have also been incorporated into the revised paper.

  1. W1 & Q1: Potential data leakage across dataset splits. Thanks for raising this important concern. Our dataset was partitioned at the collection level. Specifically, we assigned unique identifiers to all collections across all data sources and then randomly divided them into training and test sets. All images and pairwise comparisons belonging to the same collection were assigned to the same subset. Since each pair is generated exclusively within its corresponding collection, and each image belongs to only one collection, there is no overlap of collections, images, or preference pairs between the training and test sets. Importantly, this random collection-level partitioning preserves the original data distribution across subsets, with all data sources represented in each, thereby maintaining a distribution similar to that of real-world user albums. The detailed statistics are provided below:

    SplitPairsCollectionsImages
    Training33,9656,69932,358
    Test6,7521,3706,675
    Total40,7178,06939,033

    Regarding SkillOpt, its validation set was randomly sampled from 5% of the training collections and held out during skill optimization. Both the SkillOpt optimization and validation data were exclusively drawn from the training split and were therefore strictly disjoint from the held-out test collections. Consequently, no test data are involved in any stage of SkillOpt.

    The corresponding revisions have been incorporated into Appendix D.1 (Page 21, Line 1092--1093), with the modified text highlighted in blue.

  2. W2 & Q2: Annotation reliability and confidence-based filtering. Thanks for raising this important concern. We agree that evaluating annotation reliability and benchmark difficulty requires examining both annotator agreement and model performance on unfiltered comparisons.

    First, we have added a pairwise inter-annotator agreement matrix in Figure 5 of the revised manuscript to illustrate the consistency among different annotators.

    To further characterize the original annotations before confidence-based filtering, we summarize the distribution of annotator vote splits across all image pairs:

    Vote SplitPercentage
    10:06%
    9:112%
    8:219%
    7:323%
    6:426%
    5:514%

    The distribution of vote splits indicates that annotators do not always reach a unanimous agreement when evaluating the same image pair. This variation reflects the inherently subjective nature of aesthetic judgments, as individual aesthetic preferences and interpretations of visual content may differ across annotators, even when evaluating the same images.

    Second, to investigate the effect of confidence-based filtering, we have reported additional experimental results using both filtered and unfiltered training and test sets. All four comparisons use the same Qwen3.5-9B backbone with SFT. The results are presented below:

    Training SetTest SetNatural AccNatural F1Editing AccEditing F1Overall AccOverall F1
    UnfilteredUnfiltered65.54%65.49%75.02%74.90%69.82%69.74%
    FilteredUnfiltered66.17%66.14%76.10%76.04%70.65%70.65%
    UnfilteredFiltered70.69%70.62%75.82%75.68%73.03%72.93%
    FilteredFiltered72.00%71.99%76.99%76.92%74.27%74.27%

    As shown above, models trained on the filtered data consistently outperform those trained on the unfiltered data on both filtered and unfiltered test sets. Specifically, the overall accuracy improves from 69.82% to 70.65% on the unfiltered test set and from 73.03% to 74.27% on the filtered test set. These results demonstrate that confidence-based filtering improves the quality of training supervision by reducing uncertainty in the annotations, enabling the model to learn more consistent aesthetic preference relationships.

    Furthermore, as reflected by the inter-annotator agreement and vote distribution, aesthetic judgments involve inherent subjectivity and disagreements among annotators. Therefore, filtering low-confidence comparisons is important for establishing more reliable preference labels and constructing a high-quality benchmark. These additional results further support the importance of confidence-based filtering in improving the reliability of preference labels and the quality of the resulting benchmark.

    The corresponding revisions have been incorporated into Table 7 (Page 19), Figure 5 (Page 17) and Appendix B.3 (Page 20, Line 1046--1057), with the modified text highlighted in blue.

  1. W3 & Q3: Training-matched baselines and the role of contextual supervision. Thanks for raising this important concern. We agree that comparisons with off-the-shelf models alone are insufficient to distinguish the benefits of our proposed method from those of ordinary task-specific fine-tuning. To address this concern, we have conducted additional comparisons with several competing models fine-tuned on the same benchmark training split, including conventional visual models (ResNet-50, ViT-B/16, and CLIP-RN50) and aesthetic-specialized models (ArtiMuse and Q-Align). All models are evaluated on the same held-out test split. The results are presented below:

    ModelNatural AccNatural F1Editing AccEditing F1Overall AccOverall F1
    ResNet-5062.78%62.74%71.08%71.05%66.56%66.53%
    ViT-B/1661.74%61.64%70.20%70.17%65.60%65.54%
    CLIP-RN5065.41%65.36%72.93%72.91%68.84%68.80%
    Artimuse67.29%67.25%75.92%75.89%71.22%71.19%
    Q-Align67.97%67.93%77.32%77.30%72.23%72.20%
    Qwen3.5-9B (Base)65.17%65.08%69.22%68.64%67.02%66.75%
    ContextAes (Ours)72.44%72.44%78.71%78.60%75.30%75.27%

    As shown above, ContextAes consistently outperforms the competing models fine-tuned on the same training split. In particular, compared with the strongest fine-tuned baseline, Q-Align, ContextAes achieves improvements of 4.47, 1.39, and 3.07 percentage points in overall accuracy on the Natural, Editing, and Overall subsets, respectively. These results indicate that the performance advantage of ContextAes cannot be attributed solely to ordinary task-specific fine-tuning, as ContextAes maintains a clear advantage over competing methods even when they are fine-tuned on the same benchmark training data.

    To distinguish the benefits of contextual supervision from those of ordinary task-specific fine-tuning, we first fine-tuned the model with full collections as input to adapt it to collection-level processing. We then evaluated it on the test set and compared its performance with that of the fine-tuned Q-Align and ArtiMuse models. For the model using full-collection inputs, the evaluation metrics were calculated by converting the predicted rankings into pairwise preferences. The results are presented below:

    ModelNatural AccNatural F1Editing AccEditing F1Overall AccOverall F1
    ContextAes (Full Collection)60.86%60.75%78.62%78.61%69.59%69.55%
    ContextAes (Pair)72.44%72.44%78.71%78.60%75.30%75.27%
    ArtiMuse67.29%67.25%75.92%75.89%71.22%71.19%
    Q-Align67.97%67.93%77.32%77.30%72.23%72.20%

    However, processing an entire collection introduces additional computational demands as the number of input images grows. In our dataset, some collections contain up to 18 images, resulting in substantially longer visual input sequences, which may place greater demands on the model's attention and cross-image information aggregation. Since 88.54% of the collections contain no more than six images and all collections in the Editing subset contain at most six images, we further restricted the evaluation to collections containing no more than six images to provide a more controlled comparison. The results are presented below:

    ModelNatural AccNatural F1Editing AccEditing F1Overall AccOverall F1
    ContextAes (Full Collection)68.15%67.86%78.62%78.61%76.39%76.36%
    ContextAes (Pair)75.84%75.77%78.71%78.60%78.10%78.00%
    ArtiMuse68.27%68.07%75.92%75.89%74.29%74.24%
    Q-Align65.99%65.90%77.32%77.30%74.90%74.88%

    The results show that restricting the evaluation to collections containing no more than six images substantially improves the performance of the full-collection model, with its overall ACC and F1 increasing from 69.59% and 69.55% to 76.39% and 76.36%, respectively. This improvement is particularly evident in the Natural subset, where its ACC and F1 increase from 60.86% and 60.75% to 68.15% and 67.86%, respectively. Nevertheless, ContextAes (Pair) still achieves better overall performance, with an F1 of 78.00%, compared with 76.36% for the full-collection model. These results indicate that the full-collection model benefits from shorter visual input sequences, although directly processing an entire collection remains more challenging than pairwise preference prediction.

Finally, to further evaluate the models' ability to capture aesthetic preferences at the collection level, we have additionally compared their Kendall's τ scores. The results are presented below:

ModelKendall's τ (Overall)Kendall's τ (Natural)Kendall's τ (Editing)Kendall's τ (≤6 Images, Overall)Kendall's τ (≤6 Images, Natural)
ContextAes (Full Collection)0.63970.28390.77390.71410.4149
ArtiMuse0.44600.24630.52550.47850.2431
Q-Align0.46150.26560.53960.49710.2845

Note: All collections in the Editing subset contain no more than six images; therefore, the Editing results are identical in the full-test and ≤6-image evaluations.

These results suggest that the full-collection model can effectively capture contextual relationships among images, particularly for relatively small collections. For collections containing no more than six images, it achieves an overall F1 of 76.36% and a Kendall's τ of 0.7141, demonstrating its ability to capture inter-image relationships and produce coherent collection-level rankings. However, its performance decreases when evaluated on the full test set, which also includes larger collections. This decline may be attributed to the increased length of visual input sequences as more images are introduced, which places greater demands on the model's attention and cross-image information aggregation. As the collection grows, the model may struggle to effectively preserve and integrate relationships across all images. This highlights a limitation of current MLLMs in processing long visual contexts and points to an important direction for future research.

The corresponding revisions have been incorporated into Table 12 and 13 (Page 30), Table 14 (Page 31) and Appendix F.1 (Page 32, Line 1710--1746), with the modified text highlighted in blue.

  1. W4: Variability and statistical reliability of ablation results. Thanks for pointing this out. We agree that reporting only point estimates is insufficient to fully characterize the variability of the ablation results, particularly when the individual training stages exhibit different effects across configurations. To address this concern, we have performed 10,000 bootstrap resampling iterations and calculated the corresponding 95% percentile confidence intervals for ACC and F1 across the Natural, Editing, and Overall. The results are presented below:

    Natural

    Training StrategyAcc (95% CI)F1 (95% CI)
    SFT72.00% [70.53, 73.44]71.99% [70.53, 73.42]
    SkillOpt69.67% [67.65, 71.66]69.59% [67.58, 71.57]
    GRPO69.41% [67.95, 70.88]69.35% [67.86, 70.81]
    SkillOpt + SFT71.78% [70.31, 73.25]71.66% [70.17, 73.13]
    SkillOpt + GRPO67.85% [66.36, 69.37]67.51% [66.01, 69.03]
    SFT + GRPO69.31% [67.81, 70.78]68.93% [67.41, 70.43]
    SkillOpt + SFT + GRPO72.44% [70.99, 73.88]72.44% [70.98, 73.87]

    Editing

    Training StrategyAcc (95% CI)F1 (95% CI)
    SFT76.99% [75.53, 78.45]76.92% [75.45, 78.40]
    SkillOpt74.30% [72.58, 75.98]73.60% [71.84, 75.30]
    GRPO74.20% [72.67, 75.72]74.19% [72.65, 75.72]
    SkillOpt + SFT77.28% [75.79, 78.78]77.13% [75.64, 78.61]
    SkillOpt + GRPO73.74% [72.15, 75.29]73.73% [72.14, 75.29]
    SFT + GRPO75.46% [73.90, 76.94]75.30% [73.75, 76.80]
    SkillOpt + SFT + GRPO78.71% [77.28, 80.18]78.60% [77.15, 80.05]

    Overall

    Training StrategyAcc (95% CI)F1 (95% CI)
    SFT74.27% [73.22, 75.33]74.27% [73.21, 75.32]
    SkillOpt72.19% [70.87, 73.50]71.84% [70.50, 73.14]
    GRPO71.59% [70.51, 72.66]71.57% [70.48, 72.64]
    SkillOpt + SFT74.29% [73.24, 75.34]74.15% [73.09, 75.21]
    SkillOpt + GRPO70.52% [69.42, 71.64]70.40% [69.29, 71.52]
    SFT + GRPO72.11% [71.03, 73.16]71.85% [70.77, 72.91]
    SkillOpt + SFT + GRPO75.30% [74.27, 76.32]75.27% [74.25, 76.29]

    The results suggest that the contributions of the three training stages are not simply additive. Different combinations may result in overlapping benefits or incompatibilities between stages, leading to limited or even negative improvements. In our approach, SkillOpt optimizes task-specific skills, SFT enables the model to effectively learn and utilize these skills, and GRPO further refines its aesthetic preference predictions. Through this sequential integration, the complete SkillOpt+SFT+GRPO pipeline achieves the highest ACC and F1 across all three subsets. The bootstrap confidence intervals further quantify the uncertainty of the evaluation results and, together with the consistently superior performance across all three subsets, provide additional support for the effectiveness of the complete three-stage training strategy.

    The corresponding revisions have been incorporated into Table 11 (Page 28) and Appendix E.3 (Page 31, Line 1645--1648), with the modified text highlighted in blue.

Reviewer_AqnUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ContextAes-40K is introduced and ContextAes is proposed.

Critical strengths

In this paper, ContextAes-40K is introduced and ContextAes is proposed. ContextAes-40K comprises 39,033 images organized into 8,069 contextually related collections, with pairwise aesthetic preference annotations within each collection. ContextAes is a MLLM-based aesthetic model trained through a three-stage training strategy.

Critical weaknesses
  1. Although this method has certain performance advantages, it overly relies on large models, thereby masking the innovation of the method and making it difficult to assess whether the performance improvement stems from the core innovation or the basic capabilities of the large model.
  2. Add runtime analysis to show overhead of the three stage training pipeline of ContextAes.
  3. Discuss failure cases to reveal typical model weaknesses on context dependent aesthetic tasks.
  4. Compared to GRPO, SkillOpt+GRPO will result in a performance degradation. What are the underlying reasons for this?
  5. Although there are descriptions for each stage in the methodology section, the composition of the core module is partially missing and should not be presented in the appendix.
Questions for authors

it overly relies on MLLM models. Explain the necessity and innovation of the usage.

3 comments

To reviewer AqnU: Thanks for the constructive comments. Responses to specific comments are given below, which have also been incorporated into the revised paper.

  1. W1: Reliance on large models and the unclear contribution of the proposed method. Thanks for the concern. We acknowledge that the performance of ContextAes benefits from both the strong foundational capabilities of the pretrained MLLM and the innovations introduced by our proposed training method. To better distinguish these two contributions, we have conducted additional comparative experiments by fine-tuning conventional visual models and aesthetic-specialized models on the same training set. Specifically, we provide the following analyses:

    First, the choice of the underlying model plays an important role in aesthetic understanding. We have fine-tuned three conventional visual models, including ResNet-50, ViT-B/16, and CLIP-RN50, on the same training set for a fair comparison. The results are presented below:

    ModelNatural AccNatural F1Editing AccEditing F1Overall AccOverall F1
    ResNet-5062.78%62.74%71.08%71.05%66.56%66.53%
    ViT-B/1661.74%61.64%70.20%70.17%65.60%65.54%
    CLIP-RN5065.41%65.36%72.93%72.91%68.84%68.80%
    Artimuse67.29%67.25%75.92%75.89%71.22%71.19%
    Q-Align67.97%67.93%77.32%77.30%72.23%72.20%
    Qwen3.5-9B (Base)65.17%65.08%69.22%68.64%67.02%66.75%
    ContextAes (Ours)72.44%72.44%78.71%78.60%75.30%75.27%

    As shown above, even after fine-tuning on our dataset, conventional visual models achieve considerably lower performance than aesthetic-specialized models (Artimuse and Q-Align) and our proposed ContextAes. These results suggest that conventional visual models have inherent limitations in capturing aesthetic preferences, highlighting the importance of leveraging a model with stronger foundational capabilities in visual and aesthetic understanding.

    Second, the performance improvement also comes from the innovations in our training method, rather than solely from the underlying model's capabilities. To demonstrate this, we have compared the original Qwen3.5 backbone with our fully trained ContextAes model as shown in the table above. Without our proposed training pipeline, the Qwen3.5 backbone achieves an overall accuracy of 67.02% and an F1 score of 66.75%. After applying our proposed training method, ContextAes achieves 75.30% accuracy and 75.27% F1, corresponding to improvements of 8.28 and 8.52 percentage points, respectively. Furthermore, even compared with aesthetic-specialized models fine-tuned on the same training set, ContextAes consistently outperforms both Artimuse and Q-Align across the Natural, Editing, and Overall subsets. In particular, it surpasses Q-Align, the strongest competing model, by 3.07 percentage points in overall accuracy. These results demonstrate that our training method effectively enhances the aesthetic preference understanding capabilities of the underlying MLLM.

    The corresponding revisions have been incorporated into Table 1 (Page 7) and into Appendix E.1.2 (Page 26, Line 1381--1390), with the modified text highlighted in blue.

  2. W2: Runtime of the three-stage training pipeline. Thanks for the suggestion. To clarify the computational overhead of the three-stage training pipeline, we have measured the wall-clock runtime of each training stage using 8 NVIDIA H20 GPUs. The results are presented below:

    StageRuntimeProportion
    SkillOpt3 h7.5%
    SFT13 h32.5%
    GRPO24 h60.0%
    Total40 h100%

    As shown above, the complete three-stage training pipeline takes approximately 40 hours. Among the three stages, GRPO accounts for the largest proportion (60%), as it involves iterative policy optimization, repeated response generation and reward evaluation. SFT takes 13 hours (32.5%), while SkillOpt requires only 3 hours (7.5%).

    The corresponding revisions have been incorporated into Appendix D.2 (Page 22, Line 1166--1170), with the modified text highlighted in blue.

  1. W3: Typical weaknesses of baseline models in context-dependent aesthetic evaluation. Thanks for the helpful suggestion. We have added representative failure cases and corresponding analyses in Appendix E.5 (Page 32, Line 1702--1708) of the revised manuscript to further investigate the limitations of the typical weaknesses of baseline models in context-dependent aesthetic evaluation.

    Specifically, these failure cases reveal two typical weaknesses. First, baseline models may over-rely on the visual quality of individual images, such as composition and image clarity, while insufficiently considering their roles and contributions to the overall collection. Second, baseline models may struggle to capture complex relationships among images, particularly subtle semantic connections, visual consistency, and complementary relationships across a collection. These cases illustrate that understanding the aesthetic preference of an image collection requires not only evaluating the visual quality of individual images but also reasoning about their interrelationships and overall coherence.

  2. W4: Performance degradation of SkillOpt+GRPO. Thanks for raising this important point. As shown in Table 3 of the manuscript, directly combining SkillOpt with GRPO without an intermediate SFT stage indeed leads to a performance degradation. We believe this is primarily due to the lack of an intermediate adaptation stage.

    Specifically, SkillOpt optimizes explicit textual strategies rather than directly updating the model parameters. Although these optimized Skills provide additional guidance for aesthetic preference evaluation, the underlying model may not yet be sufficiently adapted to consistently follow and effectively utilize them. Applying GRPO directly at this stage may therefore result in a mismatch between the skill-guided reasoning strategies and the model's existing response behavior, preventing the model from fully benefiting from the optimized Skills.

    This also highlights the importance of the intermediate SFT stage, which serves as an adaptation process that enables the model to learn to follow and operationalize the optimized Skills before further policy optimization. As shown in Table 3, incorporating SFT between SkillOpt and GRPO increases the overall accuracy from 70.52% to 75.30%, an improvement of 4.78 percentage points. Furthermore, the complete SkillOpt+SFT+GRPO pipeline outperforms the SFT+GRPO variant by 3.19 percentage points in overall accuracy. This indicates that the optimized Skills can be more effectively utilized when the model first undergoes SFT adaptation before GRPO optimization.

  3. W5: Incomplete description of core module components. Thanks for the helpful suggestion. We agree that the composition of the core model architecture should be clearly described in the main text to facilitate understanding. In the revised manuscript, we have expanded the description of the ContextAes architecture in Section 4.1 (Page 5, Line 266--273) and moved the relevant architectural details previously presented in the appendix into the main text.

    Specifically, we have provided a more detailed description of the three main components of the Qwen3.5 backbone, including the vision module, the merger module, and the language module, clarifying their respective roles and how they interact to process visual inputs and generate aesthetic preference predictions.

  4. Q1: Necessity and novelty of using MLLMs. Thanks for raising this important concern. We use MLLMs because of the specific requirements of aesthetic preference evaluation in contextually related image collections, and our contributions extend beyond directly applying existing MLLMs to this task.

    First, unlike conventional single-image aesthetic assessment, evaluating an image collection requires understanding not only the visual quality of individual images but also their semantic connections, visual consistency, and complementary relationships. These requirements involve both fine-grained visual perception and high-level reasoning across multiple images, making MLLMs a suitable foundation for this task. As demonstrated in our response to W1, conventional visual models exhibit limitations in this setting, even after fine-tuning on our dataset.

    Second, given the limited exploration of MLLMs in this task, we have benchmarked representative existing MLLMs to evaluate their capabilities in aesthetic preference assessment within contextually related image collections. Building upon this evaluation, we have further proposed a three-stage training pipeline consisting of SkillOpt, SFT, and GRPO to enhance their task-specific capabilities. The ablation results in Table 3 further demonstrate the effectiveness of this training strategy.

AuthorsOpenReview ↗

We are wondering if the reviewers have any other questions

Dear Reviewer AqnU,

Thank you again for your constructive comments. As the author-reviewer discussion period ends in about one day, we would like to kindly ask whether our responses have addressed your concerns.

Following your suggestions, we have conducted additional experiments comparing different conventional visual models and aesthetic-specialized models, analyzed the runtime overhead of the three-stage training pipeline, and added representative failure cases and further ablation analyses. We have also analyzed the performance degradation of SkillOpt+GRPO, expanded the description of the core model architecture in the main text, and further explained the necessity and contributions of using MLLMs for this task.

We sincerely hope that these additional experiments, analyses, and clarifications have sufficiently addressed your concerns. We would be happy to clarify any remaining questions or concerns.

Best Regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses & Paper Revisions

We sincerely thank all the reviewers for their valuable and constructive feedback. We appreciate that the reviewers acknowledged that our work addresses an important and underexplored problem in aesthetic preference evaluation (Reviewers AqnU, QsfZ, Qd9F), provides valuable datasets with carefully designed annotations (Reviewers QsfZ, Qd9F), and proposes an effective MLLM-based model with promising performance and generalization ability (Reviewers AqnU, QsfZ, Qd9F). We also appreciate their recognition of the paper's clear presentation and practical significance.

We have made our best effort to address the concerns and revise the paper accordingly. The major modifications are summarized as follows:

  1. Additional evaluations and comparisons. We evaluated additional conventional visual models and aesthetic-specialized models on ContextAes-40K, further demonstrating the dataset's generalizability and the effectiveness of our proposed method.

  2. Expanded analysis of dataset construction and annotation reliability. We clarified the collection-level data splitting strategy, provided source-wise dataset statistics, and added inter-annotator agreement, vote distribution, and filtered/unfiltered evaluation results.

  3. Further investigation of contextual aesthetic preference modeling. We clarified the distinction between contextually related collections and highly similar image series, and added full-collection input comparisons and analyses under different collection sizes.

  4. Additional analysis of the proposed model and training strategy. We expanded the backbone architecture description, provided training runtime statistics, analyzed the contribution of each training stage, and reported bootstrap confidence intervals and additional failure cases.

  5. Extended model variants and practical deployment evaluations. We introduced a lightweight 2B variant for on-device inference and a score-based variant for efficient large-scale image ranking, with additional smartphone deployment and server-side efficiency evaluations.

All modifications have been highlighted in blue in the revised manuscript. Point-for-point responses to specific comments are given in the following reviewer-specific responses. We welcome any further discussions and will address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.