acceptodds
Under review as a conference paper at ICLR 2027

Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation

Abstract

Pairwise preferences and pointwise ratings are the two dominant annotation protocols in image aesthetic assessment (IAA), yet existing benchmarks adopt only one, leaving their complementarity unmeasured on the same images and dimensions. We propose an approach to expert aesthetic evaluation based on anchored preference-rating fusion. To study this approach, we construct PPAINT, a benchmark of 150 Chinese paintings spanning five aesthetic dimensions. It contains matched ratings and 45,900 pairwise judgments collected from 15 domain experts. The matched design reveals complementary strengths: preferences yield more consistent ordinal rankings, while ratings anchor scores to the expert rating scale. Our approach combines these signals into fused expert ground truth using two distinct preference-to-score constructions, which produce nearly identical scores under shared anchor constraints. Following the same preference-to-score principle, we introduce PSDISTILL, a label-free self-distillation method for VLM aesthetic scoring. It converts VLM pairwise judgments into continuous pseudo-scores via an Elo reference pool, and trains the same VLM with ranking optimization with pair weighting to produce a single-pass aesthetic scorer. Trained on a single painting category, the distilled Qwen3-VL-8B improves mean SRCC from 0.504 to 0.709 across all three categories. Among the evaluated models, it achieves a higher mean SRCC point estimate than the specialized IAA baseline and most closed-source VLMs, using a single forward pass per image. Further experiments on APDDv2 assess the training method’s generalization beyond Chinese painting. We will release the training code and provide controlled academic access to the complete PPAINT annotations and score mappings.

Primary area:
datasets and benchmarks
Keywords:
image aesthetic assessment, evaluation protocols, pairwise preferences, pointwise ratings, Chinese painting, dataset, vision-language models, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eEJoOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how annotation protocols affect aesthetic ground truth construction. The authors introduce PPAINT, a benchmark with matched expert ratings and pairwise preferences on Chinese paintings across five aesthetic dimensions. They show that preferences provide more consistent rankings, while ratings provide score calibration, and propose an anchored preference-rating fusion method to construct expert aesthetic scores. Based on this idea, they introduce PSD ISTILL, a label-free self-distillation method that converts VLM-generated preferences into continuous pseudo-scores using Elo-based aggregation and trains a single-pass aesthetic scorer. Experiments on PPAINT and APDDv2 demonstrate improvements over existing preference distillation baselines.

Critical strengths
  • The matched annotation analysis is a valuable contribution. The paper provides a careful study of rating versus preference supervision and shows their complementary roles for subjective aesthetic evaluation. This provides useful insight into how human judgments should be collected and calibrated.

  • PSD ISTILL is a reasonable and effective extension of preference-based distillation. Compared with majority-vote supervision, converting preferences into continuous scores better preserves global ranking structure and score differences, leading to consistent improvements over EvoQuality-style baselines.

Critical weaknesses
  • The generality of PSD ISTILL is not sufficiently demonstrated. Experiments mainly focus on artistic aesthetic datasets, and it remains unclear whether the proposed preference-to-score distillation framework generalizes to standard IQA benchmarks with human MOS annotations.

  • The algorithmic novelty of PSD ISTILL is relatively limited. The main technical changes over prior preference distillation methods are Elo-based pseudo-score construction and pair weighting. While effective, these improvements appear incremental compared with the stronger contribution of the annotation study.

Questions for authors
  1. Has PSD ISTILL been evaluated on standard IQA benchmarks beyond artistic domains? If not, is the intended scope limited to artistic aesthetic assessment?

  2. Since rating and preference annotations use different expert panels and fixed ordering, how do the authors disentangle protocol effects from possible panel or ordering biases?

4 comments

Thank you for the constructive review and for recognizing the value of our annotation study and the effectiveness of PSDISTILL. We appreciate your questions about generalization and annotation design. Changes and additions in the revised manuscript are marked in red.

1. Scope and generalization to IQA (Q1, W1)

PSDISTILL is designed and evaluated for artistic image aesthetic assessment (IAA). PPAINT examines expert evaluation of Chinese paintings. The APDDv2 experiments extend the evaluation to oil paintings and sketches, with separate training on each domain and testing on both.

These experiments support the use of the training method across different artistic domains. A controlled extension to standard image quality assessment (IQA) would require task-specific prompts and pseudo-labels, matched baseline training, and evaluation against human mean opinion scores (MOS). We leave this extension to future work given the time required for these additional experiments.

2. Expert panels and annotation order (Q2)

We took several steps to standardize the annotation conditions. Both protocols use the same paintings, aesthetic dimensions, and scoring rubrics. Experts were drawn from the same pool of 15 Chinese painting specialists and randomly assigned across categories, with five experts per category for each protocol. The rating and preference panels differ within each category. Before pointwise annotation, experts reviewed reference paintings outside the 150-image dataset to understand the five rating levels. These references remained available during scoring.

We collected pointwise ratings first, followed by pairwise judgments after a two-day break. We also used repeated annotations and transitivity checks for quality control, and examined ranking stability under annotator subsampling. The procedures and analyses are described in Appendices A.2 and B.5.

These measures help standardize the task and assess annotation reliability. They do not fully separate protocol effects from panel composition and fixed ordering. We therefore interpret the findings as differences observed under this collection design. This interpretation is already stated in the Limitations paragraph.

3. Contribution of PSDISTILL (W2)

The paper combines a study of expert aesthetic evaluation with a method for learning from model-generated preferences. Majority-vote supervision retains the local preference direction of each sampled pair. In our 3,000-image setup, PSDISTILL achieves a higher mean SRCC than EvoQuality-style distillation while using approximately 68% fewer offline VLM calls to construct pseudo-labels. Its reference pool aggregates sparse comparisons into global continuous pseudo-scores, whose gaps weight ranking supervision. The trained model produces five-dimensional scores from a single image.

The existing ablations from the original experiment examine both parts of this design. Using the same Qwen3-VL-8B backbone and flower-and-bird training images, PSDISTILL achieves a mean SRCC of 0.709, compared with 0.628 for EvoQuality-style distillation and 0.557 for PW-Distill. With the pseudo-scores fixed, removing pair weighting reduces mean SRCC from 0.709 to 0.660.

These comparisons provide evidence for the roles of pseudo-score construction and pair weighting. The method is described in §4, the same-backbone comparison in Appendix G.3 (Table 34), and the pair-weighting ablation in Appendix D.6 (Table 23).

PSDISTILL also reduces the offline cost of constructing pseudo-labels. For 3,000 training images, EvoQuality-style majority voting uses 32 pairwise trials per image, totaling 96,000 VLM calls. With a 50-image reference pool and 10 reference comparisons per remaining image, PSDISTILL requires 1,225 + 2,950 × 10 = 30,725 calls, including pool construction. This averages 10.24 calls per image and uses approximately 68% fewer offline VLM calls. Each call returns judgments for all five dimensions (Appendix D.1).

In an additional training run, teacher validation selected the checkpoint from the seventh training epoch before its test results were examined. This model achieves a mean SRCC of 0.6785, with paired intervals supporting gains over the base model and PW-Distill. Its mean SRCC is 0.0503 higher than EvoQuality, although that interval includes zero (Appendix G.2, Table 33).

I appreciate the authors' detailed rebuttal and additional experiments, which address several concerns and strengthen the evaluation.

Neverthelss, I agree with Reviewer 6uTt's concerns regarding experimental validity. With independent checkpoint selection, the improvement over EvoQuality is no longer statistically significant. Moreover, the comparison between rating and preference protocols remains confounded by different expert panels and fixed annotation order. I therefore lower my rating to 2.

Thank you for revisiting our work. Our additional experiments demonstrate PSDISTILL's practical value and strengthen the evidence for PPAINT's expert annotations.

1. Strong scoring performance with substantially fewer supervision calls

Our advantage combines strong scoring performance with substantially fewer supervision calls. For 3,000 training images, PSDISTILL uses only 32.0% of the offline VLM calls required by EvoQuality-style supervision: 30,725 versus 96,000, including reference-pool construction. This reduces calls for training pseudo-labels by approximately 68%. Each call covers all five dimensions, and our trained scorer requires one forward pass per image (D.1). We aggregate reference comparisons into continuous pseudo-scores and use their gaps to weight ranking supervision.

On PPAINT, teacher validation selects epoch 7 before this run's checkpoint test scores are examined. Our model achieves mean SRCC 0.6785, versus 0.5043 for the base model, 0.5567 for PW-Distill and 0.6282 for EvoQuality: relative point-estimate gains of 34.5%, 21.9% and 8.0%. The gains over the base model and PW-Distill are supported by paired 95% intervals (G.1–G.2, Table 33).

On APDDv2, our method achieves 6.6% higher mean SRCC and 5.2% higher mean PLCC than EvoQuality under shared human-rated checkpoint selection. Both methods are retrained on Oil and selected using the same validation set of 300 Oil and 300 Sketch images. With joint validation, PSDISTILL achieves 0.6520/0.6421, versus 0.6115/0.6101 for EvoQuality (E.3, Table 29).

Our validation-selected 8B model also comes within 0.0214 SRCC and 0.0034 PLCC of ArtiMuse, which is fine-tuned on a large collection of expert-annotated aesthetic data. We achieve 0.6785/0.6826, versus ArtiMuse's 0.6999/0.6860, using model-generated supervision for our additional PPAINT training. For perspective, Gemini-3.1-Pro has the highest mean SRCC among the models we evaluated, at 0.7513 (G.3, Table 34).

2. A dense expert resource validated against held-out ratings

PPAINT provides expert ratings and preferences for the same paintings across five aesthetic dimensions. We standardized evaluation criteria and assessed annotation reliability. Our procedures include reference examples in KonIQ-10k [1] and repeated-image checks in PARA [2]:

  • Task and expert preparation: Both protocols use the same 150 paintings and category-specific rubrics. Different panels are drawn from the same pool of 15 Chinese-painting specialists, with randomized assignments across categories and five experts per category/protocol. External references illustrate all five rating levels and remain accessible during scoring.
  • Viewing and rest: Timestamped interfaces enforce a 10-second initial rubric review, 5 seconds per rated image and 10 seconds per pair. Ratings precede preferences, with a two-day break.
  • Consistency checks: Each expert receives 10 repeated images and 30 repeated pairs, separated by at least 5 and 10 intervening items. Repeat-pair agreement is 78.3%–93.3%, and transitivity violations are 0.10%–4.47%. We review rankings deviating most from consensus; no expert is excluded.
  • Annotator sensitivity: We evaluate leave-one-out and all one-/three-expert subsets. Leave-one-out rank correlations are 0.975/0.953 for preferences/ratings, with Top-10 overlap of 0.856/0.707 (A.2, B.5).

Within this collection design, preferences achieve mean Kendall's W of 0.7951 versus 0.6958, a 14.3% relative difference, with positive gaps in 14/15 groups. We retain individual judgments to support further study of aesthetic variation, also documented in PARA [2].

Our additional rating-held-out experiment directly strengthens the evidence for the fused expert scores. Five-fold validation excludes all ratings of held-out paintings from anchor selection, latent estimation and sigmoid fitting, while retaining their preferences. Against withheld expert means, BTL/Elo achieve MAE 0.449/0.451 and PLCC 0.895/0.901 (C.6).

We respectfully ask you to reconsider the overall assessment based on the learning gains, 68% fewer offline supervision calls, and the expert resource supported by reliability analyses and rating-held-out validation.

[1] Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. KonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment. IEEE Transactions on Image Processing, 29:4041–4056, 2020.

[2] Yuzhe Yang et al. Personalized Image Aesthetics Assessment with Rich Attributes. CVPR, pp. 19861–19869, 2022.

Thank you again for explaining your concerns about checkpoint selection and the annotation design. If our follow-up left any point unclear, we would be happy to clarify it before the discussion closes.

Reviewer_6uTtOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the complementarity of pointwise ratings and pairwise preferences in image aesthetic assessment. It introduces PPAINT, a benchmark of 150 Chinese paintings from three categories, annotated by 15 domain experts along five aesthetic dimensions. The dataset contains 3,750 pointwise ratings and 45,900 pairwise judgments. The authors report that preferences produce more consistent ordinal rankings, whereas low-disagreement ratings can anchor the resulting scores to an interpretable 1-5 expert scale. They combine the two signals using anchored Elo and anchored Davidson Bradley-Terry constructions.

The paper then proposes PSDISTILL, a human-label-free training method. A Qwen3-VL model generates pairwise judgments over a small Elo reference pool; sparse comparisons place the remaining training images on the same latent ranking; a sigmoid maps the ranks to continuous pseudo-scores; and GRPO with a gap-weighted ranking-fidelity reward trains a single-image, five-dimensional scorer. On PPAINT, the main Qwen3-VL-8B result improves mean SRCC from 0.504 to 0.709. The paper also reports comparisons to pointwise and majority-vote distillation, evaluations against pure human-rating targets, an APDDv2 experiment, and a 32B LoRA result.

The motivating story is clear and addresses a real problem: ratings provide a scale but suffer from annotator calibration differences, while preferences provide order but no absolute scale. However, the self-distillation half does not actually use rating anchors or an external human absolute scale. Its fixed sigmoid preserves the model's Elo order and supplies continuous score gaps for training weights, but it does not by itself provide evidence that the scores are aligned with the expert rating scale; the ranking objective primarily supervises order rather than absolute calibration. Thus, the second half is more accurately described as Elo-based ranking self-distillation than as a direct extension of anchored preference-rating fusion.

Critical strengths
  1. PPAINT is a potentially valuable resource. Collecting pointwise and pairwise expert judgments for the same paintings and dimensions, retaining annotator-level information, and reporting multiple reliability diagnostics directly address an under-studied problem. The high annotation density is meaningful despite the small image count.

  2. PSDISTILL presents a coherent and efficient engineering combination. Under the same backbone and training corpus, its mean SRCC point estimate of 0.709 exceeds EvoQuality-style distillation at 0.628 and PW-Distill at 0.557; the ordering also persists against pure human-rating targets. Converting offline comparisons into a one-pass scorer is practically useful. These signals justify reevaluation under a valid selection protocol.

Critical weaknesses
  1. The main quantitative evidence is contaminated by test-set selection. Checkpoints, the main reference-pool seed, APDDv2 epochs, and the 32B checkpoint are selected using the same test/evaluation labels on which final performance is reported. Consequently, the headline result, ablations, cross-domain gains, and paired confidence intervals are optimistic by an unknown amount. This is consequential because the claimed advantages over stronger baselines are modest: the paired interval versus EvoQuality is [0.020, 0.142], while intervals versus ArtiMuse, Qwen3-VL-32B, and Claude-Sonnet-4.6 include 0.

  2. The evidence does not fully support the paper's central unifying story. The protocol comparison is confounded by panel identity, fixed order, and a roughly 15-fold annotation-time difference. The two fused-target estimators are not independent validations because they share data, anchors, and calibration. PSDISTILL itself has no rating anchor, and its ranking-fidelity reward (alongside generic format/range terms) does not identify an absolute score location aligned with the expert scale. These conceptual limitations are compounded by under-explained metric definitions: Table 2 and Figure 6 imply an aggregate Kendall's W gap of about 0.098 (Figure 6 shows 14/15 positive groups), whereas Table 6 uses a rating majority-with-tie reference and reports dimension gaps of 0.181-0.275 with 15/15 groups positive. These quantities are not directly comparable and should be reconciled; the Anchored Elo anchor target is described as a mean in the main text, while Appendix C.2 says that Davidson BT uses medians.

Questions for authors
  1. Can the authors reserve PPAINT as a one-time test set and rerun all trained models using either a genuinely independent validation split or a pre-specified, label-free checkpoint rule? Please report every tried checkpoint and reference-pool seed, multi-training-seed mean and variance, and statistical intervals that include the full model-selection process. The answer and corrected results could materially change my rating.

  2. Can the central protocol claim be tested with the same experts under counterbalanced protocol order and a comparable information or time budget? If not, will the authors narrow the claim from a protocol effect to a descriptive result under this particular collection design?

  3. Please reconcile the Kendall's W values and computation definitions in Table 2, Figure 6, and Table 6 (in particular, Table 6's rating majority-with-tie reference), and clarify whether the Anchored Elo and Davidson BT anchor targets use pointwise means or medians, respectively. Please provide the per-group values or computation code needed to reproduce these claims.

2 comments

Thank you for the careful review and for recognizing the value of our annotation study and single-image scorer. Your comments helped us clarify the selection protocol and statistical definitions. Changes and additions in the revised manuscript are marked in red.

1. Reference pools and checkpoint selection (Q1, W1)

Validation-selected PSDISTILL retains SRCC gains over the base model and PW-Distill on PPAINT and achieves higher two-domain SRCC and PLCC averages than EvoQuality on APDDv2 with checkpoint selection based on joint Oil and Sketch validation.

The main experiment's reference-pool seed was fixed in advance without using PPAINT test performance (§5.1). We reused this pool and the same 3,000 flower-and-bird images/pseudo-labels, retraining PSDISTILL for ten epochs and saving every checkpoint. A separate 300-image validation set (100/category) has no expert ratings; unfine-tuned Qwen3-VL-8B supplies teacher targets via pairwise comparisons and Elo. Selection maximizes mean SRCC against these labels. Four validation settings use this same training run:

Reference poolsValidation scopeTargets
TrainingFlower-and-bird100
TrainingAll categories (primary)300
Within validationFlower-and-bird50
Within validationAll categories150

The latter pools use 50 images/category, leaving 50 targets/category. All four selected the seventh training epoch; the choices were frozen before reading this run's checkpoint test scores (Appendix G.1, Table 31).

On the expert-labeled PPAINT test set, mean SRCC/PLCC is 0.6785/0.6826. The PPAINT comparisons retain the existing baseline checkpoints. All models use the same expert GT.

ComparisonΔ mean SRCC95% paired CI
Qwen3-VL-8B+0.1742[0.0944, 0.2532]
PW-Distill+0.1218[0.0418, 0.1942]
EvoQuality+0.0503[−0.0122, 0.1137]

Differences are PSDISTILL minus baseline. The intervals support gains over the base model and PW-Distill; EvoQuality's includes zero. We use 2,000 paired image bootstrap resamples within categories, retaining all five dimensions. They measure image-sampling uncertainty with predictions/GT fixed, excluding training and selection variability. Selection rules and full comparisons are in G.2 (Table 33); baseline point estimates are in G.3 (Table 34).

For APDDv2, both methods were retrained on Oil using existing pseudo-labels and shared an independent human-MOS validation set of 300 Oil and 300 Sketch images. Joint mean validation SRCC selects epochs 8/2 for PSDISTILL/EvoQuality. Two-domain test means are SRCC 0.6520/0.6115 and PLCC 0.6421/0.6101, respectively (Appendix E.3, Table 29). We prioritized this comparison because Oil is smaller and offers human-rated validation within the rebuttal budget.

2. Protocol comparison (Q2)

Both protocols use the same paintings, dimensions and rubrics, with quality checks and annotator subsampling (A.2, B.5). Within each category, different panels provided ratings and preferences, collected in that fixed order with a two-day break. Panel composition, order and annotation time were not independently controlled, as acknowledged in Limitations.

We therefore interpret the finding as higher ranking agreement for preferences under the collection conditions studied here.

3. Kendall’s W and anchor targets (Q3)

The mean Kendall’s W is 0.7951 for preferences and 0.6958 for ratings, with a positive difference in 14 of 15 category–dimension groups.

We recomputed W from five individual expert rankings over the same 50 paintings/category: ratings use average ranks for ties and tie-corrected W; preferences use Davidson–Bradley–Terry per expert. W is not computed against a majority-vote reference. Table 2, Figure 6 and Table 6 are aligned. Figure 6(b) reports all 15 groupwise W differences, and Table 6 summarizes them by dimension. Both anchored Elo and anchored Davidson–Bradley–Terry use median expert ratings to define anchor levels (Appendix C).

4. Expert GT and self-distillation (W2)

The two components share a preference-to-score principle, with human rating anchors used for expert GT and model-generated preferences used for self-distillation.

PSDISTILL uses continuous pseudo-score gaps to weight ranking supervision; its sigmoid alone does not establish expert-scale alignment.

Beyond cross-estimator consistency, five-fold validation withheld all ratings of held-out images from anchor selection, latent estimation and sigmoid fitting, retaining preferences. Against withheld expert means, BTL/Elo achieve MAE 0.449/0.451 and PLCC 0.895/0.901 (C.6).

For 3,000 images, pseudo-label construction including the reference pool costs 30,725 VLM calls versus 96,000 for majority voting, about 68% fewer (D.1). Inference gives five-dimensional scores in one forward pass.

Thank you again for your detailed review. In response to your question about checkpoint selection, we retrained PSDISTILL on PPAINT, selected its checkpoint using teacher-generated validation scores, and compared it with the baselines using paired bootstrap intervals. On APDDv2, we retrained both PSDISTILL and EvoQuality and used the same human-rated validation set to select their checkpoints. As the discussion draws to a close, we would welcome your thoughts on these experiments and would be happy to clarify any remaining questions.

Reviewer_8bo6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper makes two contributions around expert aesthetic evaluation ofChinese paintings. (1) PPAINT benchmark. 150 Chinese paintings (flower-and-bird, landscape,figure; 50 each) are annotated under two MATCHED protocols by 15 domainexperts (5 per category): integer 1-5 pointwise ratings (3,750 raw scores) andpairwise preferences (612 sampled pairs per category, each pair judged on allfive dimensions, yielding 45,900 judgments) along brush technique, coloration,composition, artistic conception, and overall quality. The matched designdirectly measures protocol complementarity on identical stimuli: preferencesyield more consistent ordinal rankings (Kendall's W 0.795 vs. 0.697 forratings), while ratings carry scale position. Fused expert ground truth isconstructed per category-dimension by converting preferences into latentquality scores (anchored Elo; an anchored Davidson BTL variant as anindependent check) and pinning their absolute position with low-disagreementrating anchors (levels 2-4, two per level; levels 1 and 5 excluded), followedby a shared sigmoid calibration s = 1 + 4*sigma(aq+b) fitted on the pooledanchor set. (2) PSDISTILL. A label-free self-distillation method: N=50 unlabeled imagesare exhaustively compared by the VLM itself (1,225 pairs), standard Eloconverts these judgments into continuous pseudo-scores, and GRPO with apair-weighted ranking-fidelity reward trains Qwen3-VL-8B into a single-passfive-dimensional scorer, avoiding the O(n^2) test-time cost of pairwiseinference. Trained on a single painting category, the distilled model improvesmean SRCC from 0.504 to 0.709 across all three categories, achieving a highermean SRCC point estimate than the specialized IAA baseline and mostclosed-source VLMs. Generalization of the training method is further examinedon the oil-painting and sketching subsets of APDDv2.

Critical strengths

S1. Matched dual-protocol annotation with unusually thorough consistencyverification. Because the same 150 images, the same five dimensions, and thesame expert panels supply both ratings and preferences, the paper can measurewhat no existing single-protocol IAA benchmark can: the complementarity of thetwo protocols on identical stimuli (Kendall's W 0.795 vs. 0.697). The fusedground truth is then validated more carefully than is typical for IAAdatasets: two structurally different estimators (online anchored Elo vs. MAPanchored Davidson BTL) converge to nearly identical scores (SRCC 0.9951, KStest passed on 15/15 category-dimension groups); anchored-vs-unanchoredcomparisons isolate the anchors' role to scale position rather than ranking(SRCC ~0.99 either way, but distribution alignment only with anchors); andanchor-selection sensitivity is bounded (rank SRCC >= 0.9875 under alternativeselection rules). This matters because the fused GT is the foundation of boththe evaluation targets and the training pipeline; the paper substantiates itsreliability about as well as a benchmark of this scale allows. S2. PSDISTILL resolves a real deployment tension with a clean training recipe.Pairwise VLM judgment is more reliable than direct scoring but costs O(n^2)test-time calls; the paper converts the model's own pairwise judgments intocalibrated pseudo-scores via an Elo reference pool and distills them into asingle-pass scorer through GRPO with a pair-weighted ranking reward. The gainis substantial (mean SRCC 0.504 -> 0.709, a 40.7% relative improvement) andthe ablation over pseudo-label sources is informative (calibrated Elo 0.709 >majority-vote 0.628 > PW-Distill 0.557), indicating the pseudo-scoreconstruction, not merely the RL recipe, drives the improvement.

Critical weaknesses

W1. Scale and domain narrowness limit the evidentiary reach of bothcontributions. With 150 images, 3 categories, and 5 experts per category, eachcategory-dimension ground truth rests on roughly 50 images, and the headlinecross-category generalization claim (train on one category, improve on allthree) is evaluated on these same small category splits. Credible intervalsoverlap substantially with strong closed-source models (e.g., PSDISTILL 0.709[0.637, 0.755] vs. Qwen3.6-Plus 0.694 [0.549, 0.709]), so the point-estimatesuperiority over closed-source VLMs should be stated more cautiously. Thebenchmark is best read as a controlled study of protocol fusion rather than abroadly usable IAA resource. W2. Absolute-scale validity of the fused scores is verified only indirectly,with a self-referential component. Anchor images serve simultaneously as theElo regularization targets and as the data used to fit the sigmoid calibrationparameters (a, b); no held-out images validate absolute scores. Rating levels1 and 5 are excluded from anchoring, so the scale endpoints rely on sigmoidextrapolation without direct checks. Comparisons of absolute-score fidelity(PLCC/MAE against directly collected expert ratings), as opposed to rankcorrelations, remain limited. For a paper whose central claim is that fusionyields more trustworthy ground truth, the absolute-scale evidence should meetthe same standard as the ranking evidence.

Questions for authors

Q1. [Clarification; relates to W2] The anchor images serve two rolessimultaneously: they regularize the Elo/Davidson estimates AND provide thedata for fitting the sigmoid calibration parameters (a, b). Could you reportheld-out validation of absolute scores — e.g., fit (a, b) on one split of theanchors and evaluate absolute calibration error (MAE/PLCC against the meanexpert ratings) on the remaining anchors and on non-anchor images with expertratings? This would directly test whether the fused scores are trustworthy inabsolute terms, not only in ranking. Q2. [Clarification; relates to W2] Rating levels 1 and 5 are excluded fromanchoring to avoid sigmoid boundary effects. How large is the calibrationerror for images whose expert ratings are at these extremes? Since these areprecisely the best and worst images — where miscalibration would matter mostfor downstream use — please provide evidence that sigmoid extrapolation isreliable at the endpoints (e.g., MAE stratified by true rating level). Q3. [Clarification; relates to W2] The 15 category-dimension groups are fitindependently, and cross-group comparability rests entirely on the singlepooled (a, b) calibration. Is there evidence that scores from differentgroups lie on a common scale (e.g., agreement between per-dimension fusedscores and the overall-quality dimension, or cross-group rank consistency forimages rated in multiple groups)? Q4. [Experiment request; relates to W1] The superiority over closed-sourceVLMs rests on point estimates with substantially overlapping credibleintervals (PSDISTILL 0.709 [0.637, 0.755] vs. Qwen3.6-Plus 0.694[0.549, 0.709]). Could you report paired per-image significance tests (e.g.,bootstrap over images) for the key comparisons in Table 3, so the claim"higher mean SRCC than most closed-source VLMs" can be assessed at the levelof statistical support rather than point estimates? Q5. [Experiment request; relates to W1] Given that scale is the mainlimitation, do you plan to extend expert annotation beyond 150 images (orrelease the ~9,000-image unlabeled pool together with the PSDISTILLpseudo-scores) so that follow-up work can validate the fused ground truth andthe distillation pipeline at larger scale?

3 comments

Thank you for the thoughtful review and for recognizing the value of our annotation study. We appreciate your suggestions on score calibration and model comparisons. Changes and additions in the revised manuscript are marked in red.

1. Held-out validation of absolute scores (Q1)

We ran five-fold validation, withholding all expert ratings across five dimensions for 30 paintings per fold (10 per category). We rebuilt anchors, latent qualities, and sigmoid calibration using ratings from the remaining 120 paintings. Held-out ratings were excluded throughout; existing pairwise preferences remained available. We evaluated against the withheld expert mean ratings.

ConstructionMAE ↓PLCC ↑Original-anchor MAE ↓Non-anchor MAE ↓
Anchored BTL0.4490.8950.3740.457
Anchored Elo0.4510.9010.3940.457

The last two columns group held-out predictions by original anchor status. Both methods achieve approximately 0.45 MAE and 0.90 PLCC against expert ratings withheld from every construction step. Non-anchor MAE is 0.457 for both, extending agreement beyond the original anchors. See Appendix C.6 (Table 16).

2. Calibration at the endpoints (Q2)

Using the same held-out predictions and prespecified rating intervals, BTL/Elo MAEs are 0.311/0.216 in [1,1.5)[1,1.5) and 0.512/0.425 in [4.5,5][4.5,5], containing 101 and 81 image–dimension entries. In the central interval [2.5, 3.5), they are 0.517/0.619.

These observed endpoint MAEs are comparable to or lower than those in the central interval, despite excluding rating levels 1 and 5 from anchor selection. High scores are underestimated by 0.497 points for BTL and 0.399 points for Elo on average. Appendix C.6 reports these errors and the remaining limitation (Table 17 and Figure 8).

3. Calibration across groups (Q3)

For each of the 15 category–dimension groups, we refitted the sigmoid using the other 14 groups' anchors. Latent qualities from Q1 remained fixed, so only the final calibration changed.

Mean MAE rises from 0.4486 to 0.4514 for BTL and from 0.4508 to 0.4512 for Elo: increases of only 0.0028 and 0.0004.

The additional experiment supports the stability of the shared calibration mapping; it does not establish direct score comparability across painting categories or aesthetic dimensions. Scores are interpreted under each group's rubric, with metrics computed within groups before averaging. Appendix C.6 (Table 18) reports all 15 groups.

4. Paired model comparisons (Q4)

The teacher-validation-selected model achieves mean SRCC/PLCC of 0.6785/0.6826. Selection was frozen before reading this run's checkpoint test scores. We used 2,000 paired image bootstrap resamples within categories, retaining all five dimensions and sharing samples across models.

The paired intervals support SRCC gains over the base model and PW-Distill for the validation-selected model. Its mean SRCC is 0.0503 higher than EvoQuality, although that interval includes zero.

ComparisonΔ mean SRCC95% paired CI
PSDISTILL vs. Qwen3-VL-8B+0.1742[0.0944, 0.2532]
PSDISTILL vs. PW-Distill+0.1218[0.0418, 0.1942]
PSDISTILL vs. EvoQuality+0.0503[−0.0122, 0.1137]

Among seven closed-source models, intervals are positive for Doubao-Seed-1.6, GPT-5.4-Mini, and Qwen3-VL-Plus (Appendix G.2, Table 33). Qwen3.6-Plus has mean PLCC/SRCC of 0.694/0.648 (Table 34); PSDISTILL's paired SRCC difference is 0.0304 [−0.0511, 0.1083].

Appendix G.3 (Table 34) gives the test-selected model and baseline point estimates; G.2 gives paired comparisons for that model (Table 32) and the validation-selected model (Table 33). All pointwise results use the same 150 images. The intervals describe test-image sampling uncertainty with checkpoints, predictions, and GT held fixed.

5. Data release (Q5)

Our annotation design systematically collects both pointwise ratings and dense pairwise preferences for the same paintings across five aesthetic dimensions. The protocols use shared scoring rubrics and preserve both absolute ratings and relative judgments. Across 150 paintings, we annotated 1,836 distinct image pairs, each judged by five experts on five dimensions, yielding 45,900 dimension-specific preference judgments alongside 3,750 pointwise ratings. Collecting both types of annotation across five dimensions and completing quality-control checks, including repeated annotations and transitivity checks, requires substantial expert time. We will therefore prioritize releasing the unlabeled pool of approximately 9,000 paintings together with their PSDISTILL pseudo-scores, supporting reproduction and further self-distillation studies while broader expert annotation is pursued. The annotation design, quality checks, and release plan are described in Appendices A.1–A.2.

I thank the authors for the responsive and honest rebuttal. The new experiments directly address my main concerns: the five-fold held-out validation quantifies the absolute-scale validity of the fused ground truth (MAE ≈ 0.45, PLCC ≈ 0.90), and the frozen-selection retraining quantifies the optimism of the original headline (0.709 → 0.6785). I also appreciate the explicit concessions, including that the paired interval versus EvoQuality includes zero on PPAINT and that the sigmoid alone does not establish expert-scale alignment. My main remaining regret is the benchmark's scale: with only 150 expert-annotated paintings across three categories, each category–dimension ground truth rests on roughly 50 images, which limits both the statistical resolution of the model comparisons and the benchmark's usability as a broader IAA resource. The planned release of the ~9,000-image unlabeled pool with pseudo-scores is welcome, but the promise of this well-designed annotation pipeline will only be fully realized with broader expert annotation. I maintain my rating of 3: Weak Accept.

Thank you for taking the time to read the revision and follow up. Your question about the expert rating scale led us to test the fused scores against ratings withheld from their construction. We’re glad that this experiment addressed your concern.

We appreciate your support for the planned release of the 9,000-painting pool with pseudo-scores. We also agree with your point about the scale of the expert benchmark. Collecting both ratings and pairwise preferences for more paintings in each category would help distinguish models whose performance is close, while preserving the depth of annotation you highlighted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.