- Rating
- 2(−1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how annotation protocols affect aesthetic ground truth construction. The authors introduce PPAINT, a benchmark with matched expert ratings and pairwise preferences on Chinese paintings across five aesthetic dimensions. They show that preferences provide more consistent rankings, while ratings provide score calibration, and propose an anchored preference-rating fusion method to construct expert aesthetic scores. Based on this idea, they introduce PSD ISTILL, a label-free self-distillation method that converts VLM-generated preferences into continuous pseudo-scores using Elo-based aggregation and trains a single-pass aesthetic scorer. Experiments on PPAINT and APDDv2 demonstrate improvements over existing preference distillation baselines.
Critical strengths
-
The matched annotation analysis is a valuable contribution. The paper provides a careful study of rating versus preference supervision and shows their complementary roles for subjective aesthetic evaluation. This provides useful insight into how human judgments should be collected and calibrated.
-
PSD ISTILL is a reasonable and effective extension of preference-based distillation. Compared with majority-vote supervision, converting preferences into continuous scores better preserves global ranking structure and score differences, leading to consistent improvements over EvoQuality-style baselines.
Critical weaknesses
-
The generality of PSD ISTILL is not sufficiently demonstrated. Experiments mainly focus on artistic aesthetic datasets, and it remains unclear whether the proposed preference-to-score distillation framework generalizes to standard IQA benchmarks with human MOS annotations.
-
The algorithmic novelty of PSD ISTILL is relatively limited. The main technical changes over prior preference distillation methods are Elo-based pseudo-score construction and pair weighting. While effective, these improvements appear incremental compared with the stronger contribution of the annotation study.
Questions for authors
-
Has PSD ISTILL been evaluated on standard IQA benchmarks beyond artistic domains? If not, is the intended scope limited to artistic aesthetic assessment?
-
Since rating and preference annotations use different expert panels and fixed ordering, how do the authors disentangle protocol effects from possible panel or ordering biases?
4 comments
I appreciate the authors' detailed rebuttal and additional experiments, which address several concerns and strengthen the evaluation.
Neverthelss, I agree with Reviewer 6uTt's concerns regarding experimental validity. With independent checkpoint selection, the improvement over EvoQuality is no longer statistically significant. Moreover, the comparison between rating and preference protocols remains confounded by different expert panels and fixed annotation order. I therefore lower my rating to 2.
Thank you for revisiting our work. Our additional experiments demonstrate PSDISTILL's practical value and strengthen the evidence for PPAINT's expert annotations.
1. Strong scoring performance with substantially fewer supervision calls
Our advantage combines strong scoring performance with substantially fewer supervision calls. For 3,000 training images, PSDISTILL uses only 32.0% of the offline VLM calls required by EvoQuality-style supervision: 30,725 versus 96,000, including reference-pool construction. This reduces calls for training pseudo-labels by approximately 68%. Each call covers all five dimensions, and our trained scorer requires one forward pass per image (D.1). We aggregate reference comparisons into continuous pseudo-scores and use their gaps to weight ranking supervision.
On PPAINT, teacher validation selects epoch 7 before this run's checkpoint test scores are examined. Our model achieves mean SRCC 0.6785, versus 0.5043 for the base model, 0.5567 for PW-Distill and 0.6282 for EvoQuality: relative point-estimate gains of 34.5%, 21.9% and 8.0%. The gains over the base model and PW-Distill are supported by paired 95% intervals (G.1–G.2, Table 33).
On APDDv2, our method achieves 6.6% higher mean SRCC and 5.2% higher mean PLCC than EvoQuality under shared human-rated checkpoint selection. Both methods are retrained on Oil and selected using the same validation set of 300 Oil and 300 Sketch images. With joint validation, PSDISTILL achieves 0.6520/0.6421, versus 0.6115/0.6101 for EvoQuality (E.3, Table 29).
Our validation-selected 8B model also comes within 0.0214 SRCC and 0.0034 PLCC of ArtiMuse, which is fine-tuned on a large collection of expert-annotated aesthetic data. We achieve 0.6785/0.6826, versus ArtiMuse's 0.6999/0.6860, using model-generated supervision for our additional PPAINT training. For perspective, Gemini-3.1-Pro has the highest mean SRCC among the models we evaluated, at 0.7513 (G.3, Table 34).
2. A dense expert resource validated against held-out ratings
PPAINT provides expert ratings and preferences for the same paintings across five aesthetic dimensions. We standardized evaluation criteria and assessed annotation reliability. Our procedures include reference examples in KonIQ-10k [1] and repeated-image checks in PARA [2]:
- Task and expert preparation: Both protocols use the same 150 paintings and category-specific rubrics. Different panels are drawn from the same pool of 15 Chinese-painting specialists, with randomized assignments across categories and five experts per category/protocol. External references illustrate all five rating levels and remain accessible during scoring.
- Viewing and rest: Timestamped interfaces enforce a 10-second initial rubric review, 5 seconds per rated image and 10 seconds per pair. Ratings precede preferences, with a two-day break.
- Consistency checks: Each expert receives 10 repeated images and 30 repeated pairs, separated by at least 5 and 10 intervening items. Repeat-pair agreement is 78.3%–93.3%, and transitivity violations are 0.10%–4.47%. We review rankings deviating most from consensus; no expert is excluded.
- Annotator sensitivity: We evaluate leave-one-out and all one-/three-expert subsets. Leave-one-out rank correlations are 0.975/0.953 for preferences/ratings, with Top-10 overlap of 0.856/0.707 (A.2, B.5).
Within this collection design, preferences achieve mean Kendall's W of 0.7951 versus 0.6958, a 14.3% relative difference, with positive gaps in 14/15 groups. We retain individual judgments to support further study of aesthetic variation, also documented in PARA [2].
Our additional rating-held-out experiment directly strengthens the evidence for the fused expert scores. Five-fold validation excludes all ratings of held-out paintings from anchor selection, latent estimation and sigmoid fitting, while retaining their preferences. Against withheld expert means, BTL/Elo achieve MAE 0.449/0.451 and PLCC 0.895/0.901 (C.6).
We respectfully ask you to reconsider the overall assessment based on the learning gains, 68% fewer offline supervision calls, and the expert resource supported by reliability analyses and rating-held-out validation.
[1] Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. KonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment. IEEE Transactions on Image Processing, 29:4041–4056, 2020.
[2] Yuzhe Yang et al. Personalized Image Aesthetics Assessment with Rich Attributes. CVPR, pp. 19861–19869, 2022.
Thank you again for explaining your concerns about checkpoint selection and the annotation design. If our follow-up left any point unclear, we would be happy to clarify it before the discussion closes.
Thank you for the constructive review and for recognizing the value of our annotation study and the effectiveness of PSDISTILL. We appreciate your questions about generalization and annotation design. Changes and additions in the revised manuscript are marked in red.
1. Scope and generalization to IQA (Q1, W1)
PSDISTILL is designed and evaluated for artistic image aesthetic assessment (IAA). PPAINT examines expert evaluation of Chinese paintings. The APDDv2 experiments extend the evaluation to oil paintings and sketches, with separate training on each domain and testing on both.
These experiments support the use of the training method across different artistic domains. A controlled extension to standard image quality assessment (IQA) would require task-specific prompts and pseudo-labels, matched baseline training, and evaluation against human mean opinion scores (MOS). We leave this extension to future work given the time required for these additional experiments.
2. Expert panels and annotation order (Q2)
We took several steps to standardize the annotation conditions. Both protocols use the same paintings, aesthetic dimensions, and scoring rubrics. Experts were drawn from the same pool of 15 Chinese painting specialists and randomly assigned across categories, with five experts per category for each protocol. The rating and preference panels differ within each category. Before pointwise annotation, experts reviewed reference paintings outside the 150-image dataset to understand the five rating levels. These references remained available during scoring.
We collected pointwise ratings first, followed by pairwise judgments after a two-day break. We also used repeated annotations and transitivity checks for quality control, and examined ranking stability under annotator subsampling. The procedures and analyses are described in Appendices A.2 and B.5.
These measures help standardize the task and assess annotation reliability. They do not fully separate protocol effects from panel composition and fixed ordering. We therefore interpret the findings as differences observed under this collection design. This interpretation is already stated in the Limitations paragraph.
3. Contribution of PSDISTILL (W2)
The paper combines a study of expert aesthetic evaluation with a method for learning from model-generated preferences. Majority-vote supervision retains the local preference direction of each sampled pair. In our 3,000-image setup, PSDISTILL achieves a higher mean SRCC than EvoQuality-style distillation while using approximately 68% fewer offline VLM calls to construct pseudo-labels. Its reference pool aggregates sparse comparisons into global continuous pseudo-scores, whose gaps weight ranking supervision. The trained model produces five-dimensional scores from a single image.
The existing ablations from the original experiment examine both parts of this design. Using the same Qwen3-VL-8B backbone and flower-and-bird training images, PSDISTILL achieves a mean SRCC of 0.709, compared with 0.628 for EvoQuality-style distillation and 0.557 for PW-Distill. With the pseudo-scores fixed, removing pair weighting reduces mean SRCC from 0.709 to 0.660.
These comparisons provide evidence for the roles of pseudo-score construction and pair weighting. The method is described in §4, the same-backbone comparison in Appendix G.3 (Table 34), and the pair-weighting ablation in Appendix D.6 (Table 23).
PSDISTILL also reduces the offline cost of constructing pseudo-labels. For 3,000 training images, EvoQuality-style majority voting uses 32 pairwise trials per image, totaling 96,000 VLM calls. With a 50-image reference pool and 10 reference comparisons per remaining image, PSDISTILL requires 1,225 + 2,950 × 10 = 30,725 calls, including pool construction. This averages 10.24 calls per image and uses approximately 68% fewer offline VLM calls. Each call returns judgments for all five dimensions (Appendix D.1).
In an additional training run, teacher validation selected the checkpoint from the seventh training epoch before its test results were examined. This model achieves a mean SRCC of 0.6785, with paired intervals supporting gains over the base model and PW-Distill. Its mean SRCC is 0.0503 higher than EvoQuality, although that interval includes zero (Appendix G.2, Table 33).