- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes the OASIS framework to study open-dimension image scoring. OASIS consists of two stages: definition-conditioned learning and score learning with definition augmentation, semantic score tokens, and an ordinal Wasserstein loss. The paper also introduces a dataset, MD61, which comprises 40 base dimensions and 21 held-out dimensions. The experiments show that OASIS performs well on MD61 and in other image assessment settings.
Critical strengths
-
The problem addressed is meaningful and interesting. The paper aims to address a practical limitation of fixed-objective image quality assessment: evaluation requirements can still change after a model has been trained.
-
The constructed benchmark is useful. MD61 contains 61 evaluation dimensions with detailed definitions, providing a useful basis for studying compositional generalization in visual quality assessment.
Critical weaknesses
-
The significance of the demonstrated dimension transfer remains limited. Table G.2 shows close connections between held-out and base criteria, while Appendices C.1–C.2 show that the evaluation domains and source datasets are already represented in training. The demonstrated transfer is therefore relatively constrained. Moreover, the paper does not provide case studies showing successful and failed predictions or comparisons with other methods.
-
The methodological contribution is not sufficiently separated from the training-data contribution. The experimental description does not specify whether competing methods in Table 4 receive the same MD61 supervision and have comparable training budgets. Table 6 reports the K+S+M setting only for OASIS, while competing methods use K or K+S. Although the comparison between OASIS with and without M demonstrates benefits from adding MD61 on several datasets, it does not establish whether these benefits are specific to the proposed method or would also arise for competing approaches given comparable additional supervision.
Questions for authors
-
How does OASIS compare with using the closest base definition from Table G.2 or combining the corresponding base-dimension scores according to a predefined rule, without fitting to held-out labels?
-
What threshold is used to trigger third-expert adjudication, what fraction of annotations required it, and what is the inter-rater agreement for each dimension before adjudication?
4 comments
W2:
Thank you for raising this question. We have added these experimental details to the revised manuscript.
For the competing methods in Table 4, we directly evaluated their publicly released weights. For Table 6, we use the publicly reported results from their original papers. OASIS and the competing methods differ substantially in their supervision objectives. The purpose of Table 4 is to show that generic image-quality scores have limited value as a reference for machine vision tasks, rather than to present a direct performance comparison.
Table 6 presents a direct performance comparison. In fact, Table 6 of the original manuscript already compared OASIS trained on KonIQ alone or on KonIQ+SPAQ with competing methods using the same amounts of supervision (see Section 6.5 and Table 6 of the revised manuscript). For K+S+M, the principal methods compared in Table 6 support training and scoring only for a single generic image-quality criterion and cannot use the multidimensional data in MD61. A fair comparison under K+S+M is therefore not possible.
Q2:
A third expert intervenes when the first two experts' independent ratings differ by more than 20 points. If the difference is exactly 20 points, we take the mean of the first two ratings. When adjudication is triggered, the third expert independently provides a rating, and we take the mean of all three ratings. In both cases, the final reference scores are rounded to the nearest integer (Appendix D of the revised manuscript).
We computed pooled statistics over all 33,670 valid image–dimension records, using only the raw scores from the two independent initial ratings before adjudication. This includes records that subsequently triggered third-expert adjudication and excludes the third expert's ratings and the final reference scores. The results are as follows:
| Statistic | Pooled result |
|---|---|
| Valid records with two initial ratings | 33,670 |
| PLCC between initial ratings before adjudication | 0.810 |
| SRCC between initial ratings before adjudication | 0.799 |
| Mean absolute difference between initial ratings | 11.77 points |
| Initial rating difference of 0–10 points | 17,483 records (51.92%) |
| Initial rating difference of 11–20 points | 11,262 records (33.45%) |
| Initial rating difference over 20 points, triggering third-expert adjudication | 4,925 records (14.63%) |
Thank you for the clarifications and additional experiments. The base-dimension comparisons and new-domain evaluations strengthen the evidence for meaningful compositional generalization. Together with the benchmark’s value, these results persuade me to raise my rating to weak acceptance. My main remaining concern is that methodological gains are not fully separated from differences in supervision and training budget.
Thank you for your thoughtful assessment and for raising your rating. We are pleased that you noted that the base-dimension comparisons and new-domain evaluations have strengthened the evidence for compositional generalization.
We understand your remaining concern about distinguishing methodological gains from the effects of supervision and training budget. The additional experiments help clarify these factors. In Appendix I, under the same number of training steps, multidimensional supervision achieves higher overall PLCC and SRCC than supervision on fewer dimensions. Table 7 examines the contribution of OASIS's components through ablations; removing any component reduces overall performance. Table 6 provides a complementary comparison with other IQA methods under matched training-data settings. OASIS achieves the best or second-best results on several benchmarks even when MD61-Base is not used for training. We have expanded our analysis of these results in Section 6.5 of the revised manuscript.
Taken together, these results suggest that OASIS’s performance is not explained solely by additional supervision or training data, while also showing that multidimensional supervision contributes to performance.
Thank you for your comments. We have added experiments to the revised manuscript and look forward to further discussion.
To clarify the limits of our method's generalization, we have added evaluation experiments in four entirely new domains in Appendix J of the revised manuscript. The datasets from these new domains have no overlap with MD61, the dataset introduced in our paper. Tables J.2 and J.3 provide the experimental settings, image counts, and dimension definitions, and Table J.1 reports the results. The supplementary material submitted with the original manuscript already included case studies of successful and failed predictions; we have now included these in Appendix L of the revised manuscript.
In Table J.1, underwater images and industrial surfaces test compositional generalization to unseen assessment dimensions. The underwater dimensions examine organism outlines, appendages, and body-surface textures, while the industrial-surface dimensions examine fine edges, gaps, and local scratches. These features are directly related to morphology, texture, and thin-line visibility in the MD61-Base dimensions. The knowledge acquired during training can therefore remain useful when the objects being evaluated change, and OASIS achieves better overall performance than the backbone and multiple teacher models. Galaxy and PCB images, by contrast, require generalization to unseen visual concepts: for example, analyzing the morphology of spiral arms against a low-contrast background or distinguishing real gaps in PCB traces from spurious gaps caused by imaging artifacts. It indicates that the ability to assess such visual concepts depends primarily on the backbone's existing world knowledge and cannot be acquired through fine-tuning on limited data. OASIS performs poorly in these two domains.
In response to your concern, we have added two experiments to fully substantiate the difference between using a small set of target-domain dimensions and using the full set of multidimensional dimensions. The experimental configurations and results for the four single-domain models and the full multidimensional model on the training side are provided in Appendix I and Table I.1 of the revised manuscript, and the performance curves at different numbers of training steps are shown in Figure I.1.
Training experiments. We tested the effect of training with a small set of target-domain dimensions. The results show that, under comparable data and computational resources, the model trained with multidimensional supervision consistently achieves better overall performance.
Inference experiments. Following your suggestion, we fix the weights of the full OASIS model and compare three scoring strategies on the same images. For each held-out dimension, the first strategy directly uses that dimension’s definition as input; the second selects the semantically closest base definition in advance and directly uses its predicted score; the third takes the equal-weight average of the predicted scores for the same image across the 2–4 related base dimensions. Expert ratings for the held-out dimensions in MD61-Open are used only for computing the final correlations.
The experiment covers 6,492 images and 33,670 image–dimension pairs. PLCC and SRCC are computed separately on the applicable images for each dimension, and then averaged with equal weights over the 21 dimensions.
The closest-base-definition and equal-weight-average strategies perform comparably to each other, and both fall substantially below scoring with the held-out dimension’s definition. Neither a single base dimension nor a combination of base dimensions is fully equivalent to the scoring criterion of a held-out dimension: dimensions designed for downstream tasks specify the evidence that matters, which base dimensions do not encode.