acceptodds
Under review as a conference paper at ICLR 2027

OASIS: An Open-dimension Assessment System for Image Scoring via Definition-Driven Visual Evidence Learning

Abstract

Image filtering and vision-system monitoring require application-specific criteria that may change after an assessment model has been trained. However, models trained for predefined assessment objectives offer limited flexibility when new criteria are introduced. To address this limitation, we study open-dimension image scoring, in which a model predicts a continuous score from an image and a natural-language evaluation definition. The focus is on criteria not used as scoring targets during task-specific training, with model parameters fixed at inference. To this end, we propose OASIS (Open-dimension Assessment System for Image Scoring), a two-stage framework that connects evaluation definitions, criterion-relevant visual evidence and continuous scores. Specifically, definition-conditioned descriptive supervision is first used to guide the model toward visual evidence relevant to the specified criterion. The model is then trained for scoring with semantics-preserving definition augmentation and ordinal distribution learning, which encourage reliance on definition content and account for the ordering and distance between score levels. To support training and evaluation, we construct MD61, comprising 40 base dimensions with scores and evidence descriptions curated through a Score-and-Rank Verification Process, and 21 held-out dimensions annotated by five computer vision experts. Experimental results on the held-out dimensions show that OASIS outperforms evaluated baselines, such as conventional and language-driven image quality assessment (IQA) methods, and achieves an overall PLCC/SRCC of 0.829/0.823. Further experiments demonstrate that its scores correlate with downstream vision performance under controlled image degradations and that multidimensional supervision improves cross-dataset transfer on conventional IQA benchmarks. These results support the feasibility of adapting image assessment to newly specified criteria through natural-language definitions without retraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Quality Assessment, Open-Dimension Scoring, Visual Evidence Learning, Criterion-Conditioned Scoring, Ordinal Calibration, Task-Oriented Quality

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_hUWnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the OASIS framework to study open-dimension image scoring. OASIS consists of two stages: definition-conditioned learning and score learning with definition augmentation, semantic score tokens, and an ordinal Wasserstein loss. The paper also introduces a dataset, MD61, which comprises 40 base dimensions and 21 held-out dimensions. The experiments show that OASIS performs well on MD61 and in other image assessment settings.

Critical strengths
  1. The problem addressed is meaningful and interesting. The paper aims to address a practical limitation of fixed-objective image quality assessment: evaluation requirements can still change after a model has been trained.

  2. The constructed benchmark is useful. MD61 contains 61 evaluation dimensions with detailed definitions, providing a useful basis for studying compositional generalization in visual quality assessment.

Critical weaknesses
  1. The significance of the demonstrated dimension transfer remains limited. Table G.2 shows close connections between held-out and base criteria, while Appendices C.1–C.2 show that the evaluation domains and source datasets are already represented in training. The demonstrated transfer is therefore relatively constrained. Moreover, the paper does not provide case studies showing successful and failed predictions or comparisons with other methods.

  2. The methodological contribution is not sufficiently separated from the training-data contribution. The experimental description does not specify whether competing methods in Table 4 receive the same MD61 supervision and have comparable training budgets. Table 6 reports the K+S+M setting only for OASIS, while competing methods use K or K+S. Although the comparison between OASIS with and without M demonstrates benefits from adding MD61 on several datasets, it does not establish whether these benefits are specific to the proposed method or would also arise for competing approaches given comparable additional supervision.

Questions for authors
  1. How does OASIS compare with using the closest base definition from Table G.2 or combining the corresponding base-dimension scores according to a predefined rule, without fitting to held-out labels?

  2. What threshold is used to trigger third-expert adjudication, what fraction of annotations required it, and what is the inter-rater agreement for each dimension before adjudication?

4 comments

Thank you for your comments. We have added experiments to the revised manuscript and look forward to further discussion.

W1:

To clarify the limits of our method's generalization, we have added evaluation experiments in four entirely new domains in Appendix J of the revised manuscript. The datasets from these new domains have no overlap with MD61, the dataset introduced in our paper. Tables J.2 and J.3 provide the experimental settings, image counts, and dimension definitions, and Table J.1 reports the results. The supplementary material submitted with the original manuscript already included case studies of successful and failed predictions; we have now included these in Appendix L of the revised manuscript.

In Table J.1, underwater images and industrial surfaces test compositional generalization to unseen assessment dimensions. The underwater dimensions examine organism outlines, appendages, and body-surface textures, while the industrial-surface dimensions examine fine edges, gaps, and local scratches. These features are directly related to morphology, texture, and thin-line visibility in the MD61-Base dimensions. The knowledge acquired during training can therefore remain useful when the objects being evaluated change, and OASIS achieves better overall performance than the backbone and multiple teacher models. Galaxy and PCB images, by contrast, require generalization to unseen visual concepts: for example, analyzing the morphology of spiral arms against a low-contrast background or distinguishing real gaps in PCB traces from spurious gaps caused by imaging artifacts. It indicates that the ability to assess such visual concepts depends primarily on the backbone's existing world knowledge and cannot be acquired through fine-tuning on limited data. OASIS performs poorly in these two domains.

Q1:

In response to your concern, we have added two experiments to fully substantiate the difference between using a small set of target-domain dimensions and using the full set of multidimensional dimensions. The experimental configurations and results for the four single-domain models and the full multidimensional model on the training side are provided in Appendix I and Table I.1 of the revised manuscript, and the performance curves at different numbers of training steps are shown in Figure I.1.

Training experiments. We tested the effect of training with a small set of target-domain dimensions. The results show that, under comparable data and computational resources, the model trained with multidimensional supervision consistently achieves better overall performance.

Inference experiments. Following your suggestion, we fix the weights of the full OASIS model and compare three scoring strategies on the same images. For each held-out dimension, the first strategy directly uses that dimension’s definition as input; the second selects the semantically closest base definition in advance and directly uses its predicted score; the third takes the equal-weight average of the predicted scores for the same image across the 2–4 related base dimensions. Expert ratings for the held-out dimensions in MD61-Open are used only for computing the final correlations.

The experiment covers 6,492 images and 33,670 image–dimension pairs. PLCC and SRCC are computed separately on the applicable images for each dimension, and then averaged with equal weights over the 21 dimensions.

Scoring method at inferenceOverall PLCCOverall SRCC
Single semantically closest base definition0.6790.674
Equal-weight average of predicted scores across all related base dimensions0.6730.670
Direct input of the target-dimension definition0.8290.823

The closest-base-definition and equal-weight-average strategies perform comparably to each other, and both fall substantially below scoring with the held-out dimension’s definition. Neither a single base dimension nor a combination of base dimensions is fully equivalent to the scoring criterion of a held-out dimension: dimensions designed for downstream tasks specify the evidence that matters, which base dimensions do not encode.

W2:

Thank you for raising this question. We have added these experimental details to the revised manuscript.

For the competing methods in Table 4, we directly evaluated their publicly released weights. For Table 6, we use the publicly reported results from their original papers. OASIS and the competing methods differ substantially in their supervision objectives. The purpose of Table 4 is to show that generic image-quality scores have limited value as a reference for machine vision tasks, rather than to present a direct performance comparison.

Table 6 presents a direct performance comparison. In fact, Table 6 of the original manuscript already compared OASIS trained on KonIQ alone or on KonIQ+SPAQ with competing methods using the same amounts of supervision (see Section 6.5 and Table 6 of the revised manuscript). For K+S+M, the principal methods compared in Table 6 support training and scoring only for a single generic image-quality criterion and cannot use the multidimensional data in MD61. A fair comparison under K+S+M is therefore not possible.

Q2:

A third expert intervenes when the first two experts' independent ratings differ by more than 20 points. If the difference is exactly 20 points, we take the mean of the first two ratings. When adjudication is triggered, the third expert independently provides a rating, and we take the mean of all three ratings. In both cases, the final reference scores are rounded to the nearest integer (Appendix D of the revised manuscript).

We computed pooled statistics over all 33,670 valid image–dimension records, using only the raw scores from the two independent initial ratings before adjudication. This includes records that subsequently triggered third-expert adjudication and excludes the third expert's ratings and the final reference scores. The results are as follows:

StatisticPooled result
Valid records with two initial ratings33,670
PLCC between initial ratings before adjudication0.810
SRCC between initial ratings before adjudication0.799
Mean absolute difference between initial ratings11.77 points
Initial rating difference of 0–10 points17,483 records (51.92%)
Initial rating difference of 11–20 points11,262 records (33.45%)
Initial rating difference over 20 points, triggering third-expert adjudication4,925 records (14.63%)

Thank you for the clarifications and additional experiments. The base-dimension comparisons and new-domain evaluations strengthen the evidence for meaningful compositional generalization. Together with the benchmark’s value, these results persuade me to raise my rating to weak acceptance. My main remaining concern is that methodological gains are not fully separated from differences in supervision and training budget.

Thank you for your thoughtful assessment and for raising your rating. We are pleased that you noted that the base-dimension comparisons and new-domain evaluations have strengthened the evidence for compositional generalization.

We understand your remaining concern about distinguishing methodological gains from the effects of supervision and training budget. The additional experiments help clarify these factors. In Appendix I, under the same number of training steps, multidimensional supervision achieves higher overall PLCC and SRCC than supervision on fewer dimensions. Table 7 examines the contribution of OASIS's components through ablations; removing any component reduces overall performance. Table 6 provides a complementary comparison with other IQA methods under matched training-data settings. OASIS achieves the best or second-best results on several benchmarks even when MD61-Base is not used for training. We have expanded our analysis of these results in Section 6.5 of the revised manuscript.

Taken together, these results suggest that OASIS’s performance is not explained solely by additional supervision or training data, while also showing that multidimensional supervision contributes to performance.

Reviewer_kQAJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies open-dimension image scoring, where a model receives an image and a natural-language evaluation definition and predicts a continuous score without updating its parameters for the new criterion. The proposed OASIS framework combines definition-conditioned description training with score-distribution supervision, definition augmentation, and an ordinal loss. The authors also introduce MD61, containing 40 base dimensions for training and 21 held-out dimensions for evaluation. Experiments show improved agreement with expert ratings, correlations with downstream task performance under controlled degradations, and improved transfer to several conventional IQA datasets.

Critical strengths
  1. The application is useful. Allowing users to specify assessment criteria through natural-language definitions could reduce the need to collect labels and adapt a model whenever the scoring objective changes. MD61 provides a concrete training and evaluation resource for studying this setting.
  2. The evaluation includes several relevant checks. Held-out dimensions are assessed against expert ratings, and downstream task models provide an additional reference. The full-prompt, definition-only, name-only, and shared-MOS comparisons also provide useful evidence that the supplied definition affects scoring performance.
Critical weaknesses
  1. The additional contribution beyond basic supervised adaptation is limited and insufficiently characterized. In Table 7, the score-only baseline already achieves 0.812/0.802 PLCC/SRCC, compared with 0.829/0.823 for the complete framework. Thus, much of the improvement over the untuned backbone is obtained without the proposed additional components. Furthermore, while Table 4 includes direct scoring with Qwen3-VL-8B, it does not evaluate the annotation teachers on MD61-Open. Evaluating at least one teacher with the same definitions would help clarify whether the main contribution is improved assessment capability or the distillation of existing capability into a smaller, specialized scorer.
  2. The benefit specifically attributable to multidimensional training remains unclear. Table 6 shows that adding MD61 improves several cross-dataset results, but this also changes the amount and source distribution of the training images and supervision. It therefore does not isolate the effect of learning multiple assessment dimensions jointly. There is no controlled comparison with separately fine-tuned scorers, nor evidence showing how much target-dimension supervision the shared model saves. I recognize that fine-tuning with target-dimension labels is a different setting from the proposed zero-shot evaluation. Nevertheless, a limited comparison with this practical alternative, or a target-label efficiency study, would clarify when the proposed framework is preferable to straightforward task-specific fine-tuning.
  3. The benchmark does not sufficiently distinguish criterion-specific generalization from transfer of closely related quality judgments. The paper appropriately limits its scope to compositions of learned visual concepts. However, Table G.2 shows substantial proximity between held-out and base criteria—for example, lane-marking traceability versus thin-linear-structure traceability. In addition, MD61-Open contains progressively degraded versions of 1,082 source images, so several criteria may share a strong degradation-severity signal. The prompt ablations support definition dependence, but pooled correlations do not fully establish whether the model follows meaningfully different assessment rules. A targeted test in which changing the criterion should reverse the ranking of the same image pair would provide more direct evidence.
Questions for authors
  1. How does at least one annotation teacher perform on MD61-Open when given the same complete definitions? Does OASIS offer an accuracy or efficiency advantage over this alternative?
  2. Is there a controlled experiment isolating multidimensional supervision from additional training data? Alternatively, how does direct target-specific fine-tuning compare with OASIS when only a small number of target-dimension labels are available?
  3. Can you provide examples or a targeted evaluation where the same image pair should receive opposite rankings under different criteria, demonstrating that the model follows the criterion beyond shared degradation cues?
1 comment

Thanks for your constructive comments. We have added experiments to address your concerns.

W1:

Section 6.6 and Table 7 of the revised manuscript compare the score-only baseline with full OASIS and report both the progressive addition of training components and ablations that remove individual components from the full configuration.

Table 7 now reports leave-one-out removal of each component. Removing any single component lowers overall performance consistently, with the largest drop when visual-evidence supervision (Stage 1) is removed. This is consistent with its role: it shapes the representation on which the subsequent scoring stage operates, so that differences in visual evidence can be converted into score differences more effectively. Definition augmentation improves generalization across dimension definitions, and ordinal score modeling provides additional supervision on the relative ordering of images during training; removing either also lowers overall performance.

W1&Q1:

We have added direct-scoring results for the three teacher models using the same MD61-Open images and complete definitions, together with comparisons against OASIS, in Section 6.3 and Table 4 of the revised manuscript.

The results show that OASIS achieves more accurate scoring than each of the teacher models.

W2&Q2:

We have added metrics throughout training for four models trained with a small number of target-dimension labels, as well as for models trained on Base subsets of different sizes. The experimental configurations, domain-specific results, and overall results are provided in Appendix I and Table I.1 of the revised manuscript. Figure I.1 presents performance curves at different numbers of training steps.

In Appendix I, each configuration is trained independently using data of a different composition. Models trained with multidimensional supervision perform better overall than those trained with a small number of target-dimension labels.

In particular, the Base 5% configuration uses a training set comparable in size to those of the single-domain configurations and achieves better overall results. With larger training sets, OASIS's overall performance improves further.

W3&Q3:

We included a similar experiment in the supplementary material accompanying our original submission. In the revision, we further evaluate preference reversals on undegraded original images. The experimental setup, results, and examples are provided in Appendix G, Table G.1, and Figure G.1 of the revised manuscript.

This experiment keeps the image pair fixed and changes only the evaluation criterion. OASIS with complete dimension definitions achieves 70.23% joint ordering accuracy, demonstrating that its scores are not limited to generic image quality and that it has learned the distinct constraints imposed by different dimension definitions.

Reviewer_bw2fOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OASIS, a unified framework for open-dimension image scoring, where image quality is assessed according to user-specified natural-language definitions rather than predefined evaluation criteria. The authors propose a two-stage training strategy that combines definition-conditioned description learning, definition augmentation, and ordinal score modeling. They also construct MD61, covering 40 training dimensions and 21 held-out evaluation dimensions across diverse visual scenarios. Experimental results demonstrate promising performance on unseen assessment dimensions and conventional IQA benchmarks.

Critical strengths
  1. The paper addresses an interesting and practically relevant problem. Image quality is inherently dependent on application scenarios and task requirements, while conventional IQA methods primarily focus on fixed assessment objectives. Extending image assessment toward flexible, definition-driven criteria represents a meaningful direction for the community. I particularly appreciate the motivation behind open-dimension scoring and its potential applications in downstream vision systems.

  2. This work put considerable effort into constructing MD61, including multidimensional definitions, scoring annotations, and expert evaluations. The experiments cover several distinct visual domains, and the inclusion of criterion-dependent ordering analysis and downstream task alignment provides useful perspectives beyond conventional IQA evaluation.

Critical weaknesses
  1. Although the paper introduces open-dimension image scoring as a new assessment paradigm, its evaluation still relies heavily on conventional IQA metrics such as PLCC and SRCC. These metrics measure scoring consistency but do not fully reflect whether OASIS can effectively support task-specific decisions. In particular, Table 5 only demonstrates correlations with downstream task performance, where the improvements over existing methods are relatively modest in some scenarios. A more convincing evaluation would examine whether OASIS can improve downstream performance when used as a quality reward or selection criterion in image enhancement or restoration pipelines.

  2. The term "open-dimension" appears somewhat overstated. The proposed setting resembles open-vocabulary image quality assessment, where natural-language definitions specify evaluation attributes beyond a predefined set. Moreover, the held-out dimensions primarily involve compositions of visual concepts already covered during training. Therefore, the experiments mainly demonstrate compositional generalization to unseen assessment criteria, rather than unrestricted open-dimension assessment. The authors should clarify this distinction and appropriately scope their claims.

  3. A central claim of OASIS is its ability to identify visual evidence relevant to different evaluation definitions. However, it remains unclear whether the model genuinely relies on criterion-specific evidence or is still strongly influenced by generic image quality. This concern is particularly relevant because MD61-Open contains multiple synthetic degradation levels, which may introduce strong correlations between overall degradation severity and dimension-specific scores. Although the supplementary material includes criterion-dependent ordering experiments, the relatively limited joint ordering accuracy suggests that distinguishing different assessment preferences remains challenging. More controlled experiments are needed to substantiate this central claim.

  4. Additionally, the paper does not clearly state whether the complete MD61 dataset and annotations will be publicly released. MD61-Open relies on five experts, with only two independently rating each sample. However, inter-annotator agreement, disagreement statistics, and domain-specific expertise are insufficiently documented, raising concerns about annotation reliability.

  5. Moreover, the practical necessity of unifying heterogeneous domains such as medical imaging, remote sensing, and autonomous driving remains unclear, without comparisons against domain-specific fine-tuning. Table 6 also shows inconsistent benefits from additional MD61 supervision, including a noticeable performance drop on KonIQ-10k. The use of GPT-generated descriptions further raises concerns about comparison fairness, requiring clarification of training budgets, baseline protocols, and potential data overlap.

minor weakness: The overall presentation could be improved. The writing is somewhat fragmented, with important implementation details and experimental settings scattered across the main paper, appendices, and supplementary material, making the paper less accessible than necessary. Additionally, the typography and formatting appear visually inconsistent, which affects readability. I encourage the authors to improve the organization and presentation and verify the manuscript formatting against the official conference template

Questions for authors
  1. Further evidence is needed to demonstrate that OASIS learns criterion-specific visual evidence rather than relying on generic image quality. Its practical utility should also be validated through actual downstream task improvements, rather than correlations alone.

  2. The reliability of MD61-Open annotations should be supported by inter-annotator agreement, disagreement statistics, and adjudication frequency. The authors should also clarify the domain expertise of annotators and the public release plan for the complete dataset and annotations.

  3. A controlled comparison with domain-specific fine-tuning is needed to justify unified multidimensional training. The training protocols in Table 6, particularly additional descriptive supervision, computational budgets, and potential data overlap, should also be clarified.

4 comments

Thanks for the detailed and constructive assessment. We have addressed each with added experiments and expanded documentation. Point-by-point responses follow.

W3&Q1:

We agree that image degradation may influence the results on MD61-Open. Following your suggestion, we designed and conducted a preference-reversal experiment using the undegraded original images in MD61-Open. The experimental design, results, and examples are provided in Appendix G, Table G.1, and Figure G.1 of the revised manuscript.

With the influence of synthetic image degradation removed, OASIS achieves 70.23% joint ordering accuracy. This demonstrates that its scoring goes beyond generic image quality and captures the distinct constraints imposed by different dimension definitions.

W1&Q1:

We evaluated the sample-selection use case you suggested on two common downstream tasks: road object detection and remote-sensing semantic segmentation. The experimental setup and the road object detection results are given in Appendix H and Table H.1 of the revised manuscript, and the remote-sensing segmentation results in Table H.2.

Specifically, we performed image selection on a road object detection dataset comprising BDD100K and ACDC images, and on the LoveDA remote-sensing dataset. The selection criteria cover all MD61-Open dimensions in the corresponding domain. Each image receives an overall score based on the corresponding definitions, and a prescribed number of images are retained according to these scores. The experiment examines whether task-directed selection by OASIS improves performance over other selection methods at the same dataset size and computational cost.

We compared four settings: no filtering, random selection at the same retention rate, selection by Qwen3-VL-8B-Instruct, and selection by OASIS. Under a fixed training budget, we observe measurable gains in downstream performance. The advantage is more pronounced in the early training regime: after 500 updates, OASIS-based selection raises road detection mAP@50:95 from 12.75 to 14.09 (+1.34) and remote-sensing segmentation mIoU from 42.64 to 43.88 (+1.24) over Qwen-based selection. These results indicate that OASIS can select high-utility data from a large pool according to task requirements and improve downstream model performance. This suggests practical value for large-scale data selection.

W2:

We agree with this characterization and have revised the wording in the updated manuscript. The distinction is clarified below:

Compositional generalization to unseen assessment dimensions means using general, low-level visual concepts learned during training to score images under new dimension definitions.

Unseen visual concepts refer to abstract concepts or logical features that were entirely absent from training.

To address your concern and clarify our position, we have added evaluations in four new domains in Appendix J of the revised manuscript. The experimental settings, image counts, and dimension definitions are provided in Tables J.2 and J.3, and the results are reported in Table J.1.

In Table J.1, the underwater and industrial-surface evaluations test compositional generalization to unseen assessment dimensions. The underwater dimensions examine organism outlines, appendages, and body-surface textures, while the industrial-surface dimensions examine fine edges, gaps, and local scratches. These cues are directly related to morphology, texture, and thin-line visibility in the MD61-Base dimensions. The learned concepts therefore remain useful when the assessment target changes. By contrast, the galaxy and PCB evaluations require generalization to unseen visual concepts, such as analyzing the morphology of spiral arms against a low-contrast background or distinguishing real gaps in PCB traces from spurious gaps caused by imaging artifacts. We believe that the ability to assess such visual concepts depends primarily on the backbone's existing world knowledge and cannot be acquired through fine-tuning on limited data. These cases therefore fall outside the generalization scope of OASIS.

W4&Q2:

We have added the relevant details to the appendices of the revised manuscript and plan to release the annotation records and the complete dataset upon acceptance. We address your questions below.

All five experts who annotated MD61-Open are computer vision researchers, with research backgrounds spanning image classification, object detection, and semantic segmentation. Before formal annotation, all received training on the dimension-specific scoring criteria. Each image–dimension record was first rated independently on a 0–100 scale by two experts who could not see each other's ratings. If their scores differed by no more than 20 points, the mean of the two scores was used. If the difference exceeded 20 points, a third expert independently provided a score, and the mean of all three scores was used. All final reference scores were rounded to the nearest integer.

Disagreement between the initial ratings and the frequency of adjudication are reported in Appendix D and Table D.1 of the revised manuscript. The correlations and mean absolute errors between each annotator's ratings and the final reference scores are reported in Table D.2.

Upon acceptance, we will release the complete MD61 dataset. For MD61-Base, this includes the image indices, scores for the 40 base dimensions, and evidence descriptions. For MD61-Open, this includes the image indices, full definitions of the 21 dimensions, and, for each record, the two experts' initial ratings, the third expert's rating when adjudication was triggered, and the final reference score. Images will be provided through source indices so that they can be obtained under the licenses of the original datasets.

W5&Q3:

Following your recommendations, we have added data ablation experiments and incorporated the corresponding details into the revised manuscript.

The configurations and results for the four single-domain scoring models and the different-sized MD61-Base subsets are provided in Appendix I and Table I.1. Performance curves at different numbers of training steps are shown in Figure I.1.

Rationale for unified multidimensional training. In Appendix I, Base 5% uses a training set comparable in size to those of the four single-domain configurations, yet achieves better overall performance. OASIS improves further as the training data scale increases.

In Table 6, adding MD61-Base improves results on benchmarks such as KADID, PIPAL, and CSIQ, while performance on KonIQ declines. It indicates that the additional supervision changes what the model attends to. The additional MD61-Base supervision dilutes the subjective preferences represented in KonIQ, reducing the model's fit to KonIQ.

Additional descriptive supervision, computational budgets, and potential data overlap. In Stage 1, OASIS(K) and OASIS(K+S) use the training images from KonIQ and KonIQ+SPAQ, respectively. The Stage-1 quality descriptions for these configurations were generated by GPT-5.4. The K+S+M configuration retains KonIQ-10k and SPAQ and adds the remaining multidimensional supervision from MD61-Base. Because MD61-Base already contains some KonIQ-10k and SPAQ data, these overlapping records were removed from the additional MD61-Base data in the K+S+M configuration. All training-data configurations use the same number of training steps and the same batch size.

Minor weakness:

We have checked these issues in the revised manuscript and moved some important material from the supplementary material into the appendices.

I appreciate the authors' efforts in addressing my concerns through the additional experiments and clarifications. Most of my major concerns have been adequately resolved in the revised manuscript. But I still have some reservations regarding the rationale behind selecting these particular four evaluation domains, as their selection appears somewhat empirical. Nevertheless, I believe that exploring definition-driven, open-dimension image assessment is a meaningful research direction. More importantly, the proposed benchmark could make a valuable contribution to the IQA community by facilitating further research beyond predefined quality dimensions. I look forward to the public release of the MD61 dataset. Overall, I maintain my positive rating and wish the authors the best of luck.

Thank you for your positive assessment of our work. We selected autonomous driving, depth estimation , medical imaging, and remote sensing as our four evaluation domains based primarily on the extent of prior work in each domain, the diversity of available datasets, and the distinct visual characteristics of the images. We plan to extend our evaluation to additional domains in future work.

We are greatly encouraged that you noted that our proposed benchmark could make a valuable contribution to the IQA community by facilitating further research beyond predefined quality dimensions. We will make the MD61 dataset, models, code, and related resources openly available, and we hope they will provide valuable insights to the research community.

Reviewer_t5peOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OASIS, an open-dimension image scoring framework that evaluates images according to user-defined natural language criteria. It introduces a two-stage training strategy combining definition-driven visual evidence learning, definition augmentation, and ordinal score modeling, together with a new benchmark, MD61, covering 61 assessment dimensions. Experiments evaluate generalization to held-out dimensions, downstream task relevance, and transfer to conventional image quality assessment.

Critical strengths

1.The paper addresses flexible image scoring beyond predefined quality dimensions, which is potentially useful for diverse vision applications. 2. The two-stage training strategy and MD61 benchmark provide a structured approach to definition-conditioned image scoring. 3. OASIS achieves encouraging correlations with expert ratings on held-out assessment dimensions and shows potential for downstream task evaluation.

Critical weaknesses
  1. The novelty of OASIS relative to prompt-conditioned IQA and open-world assessment methods is not sufficiently established, particularly regarding its definition-driven learning mechanism.
  2. The held-out dimensions appear to involve combinations of visual concepts observed during training, leaving the extent of generalization to genuinely novel concepts unclear.
  3. The reliability of synthetic annotations, the effectiveness of Rank-Gate, and the individual contributions of the proposed components require more comprehensive validation.
Questions for authors

Could the authors clarify how OASIS differs substantially from recent related methods, particularly PromptIQA, PPA, and open-world image aesthetic assessment? How do the authors distinguish compositional generalization to unseen assessment dimensions from generalization to genuinely unseen visual concepts? Finally, could they provide stronger validation of synthetic annotation quality and more controlled ablations to isolate the contributions of visual evidence supervision, definition augmentation, ordinal score modeling, and Rank-Gate?

1 comment

Thanks for your constructive comments and positive assessment of our method.

W1&Q1:

The three works you mentioned differ from ours in two respects: how a new criterion is conveyed to the model, and which training signal establishes the link between criteria and images.

  • PromptIQA: Conveys a new criterion through a few exemplar image–score pairs as visual in-context conditioning, and scores by visual-prompt regression; the criterion itself is expressed by the support examples rather than by a textual rule.
  • PPA: Uses CLIP as its backbone, with scoring conditioned on short text prompts which provides a brief description rather than a structured evaluation rule, and the visual representation is that of a CLIP encoder rather than a VLM.
  • Open-World IAA: Addresses aesthetic assessment, with its scope centered on aesthetic quality rather than on scoring for non-aesthetic tasks.
  • OASIS (our method): It requires no additional scoring exemplars, supports complex scoring criteria specified through natural-language definitions, and covers the perceptual-quality and task-utility dimensions studied in this work.

We have added a dedicated comparison with them in Section 2.3 of the revised manuscript.

W2&Q2:

  • Compositional generalization to unseen assessment dimensions refers to the model using general, low-level visual concepts learned during training to score images under new dimension definitions.

  • Unseen visual concepts refer to abstract concepts or logical features entirely absent from training.

To address your concern and clarify our position, we have added evaluations on four new domains in Appendix J of the revised manuscript. Tables J.2 and J.3 provide the experimental settings, image counts, and dimension definitions, while Table J.1 reports the results.

In Table J.1, underwater imagery and industrial surfaces test compositional generalization to unseen assessment dimensions. The underwater dimensions examine organism outlines, appendages, and body-surface textures, while the industrial-surface dimensions examine fine edges, gaps, and local scratches. These features are directly related to morphology, texture, and thin-line visibility in the MD61-Base dimensions. The training therefore remains useful when the evaluation objects change, and OASIS achieves better overall performance than the backbone and multiple teacher models.

Galaxy and PCB images, by contrast, require generalization to unseen visual concepts: for example, analyzing the morphology of spiral arms against a low-contrast background or distinguishing real gaps in PCB traces from spurious gaps caused by imaging artifacts. OASIS degrades in these two domains. It indicats that the ability to assess such visual concepts depends primarily on the backbone's existing world knowledge and cannot be acquired through fine-tuning on limited data.

W3&Q3:

Thank you for this suggestion, which prompted us to expand the validation. We have added experiments to the revised manuscript. Section 6.2 and Table 3 validate the reliability of synthetic annotations, different teacher combinations, and Rank-Gate. Section 6.6 and Table 7 report cumulative-addition and leave-one-out ablations of the training components.

Reliability of synthetic annotations and the contribution of Rank-Gate: In Table 3, we have added comparisons using other models and model combinations on SPAQ. The results show that every model combination outperforms single-model scoring, and adding Rank-Gate yields gains for almost all scoring combinations.So averaging scores from multiple models mitigates their systematic biases, while Rank-Gate further reduces scoring contradictions, thereby improving agreement with human ratings.

Ablations and the contributions of the remaining components: Table 7 additionally reports model performance after removing each component. When visual-evidence supervision is removed, overall performance falls almost to the score-only baseline, even when the other scoring components are retained. It argues that visual-evidence supervision improves the model's understanding of dimension definitions and visual evidence, enabling subsequent definition augmentation and ordinal score modeling to translate differences in visual evidence into score differences more effectively. Definition augmentation improves generalization across dimension definitions, while components such as ordinal score modeling provide additional supervision on the ordering relationships between images during training. Removing definition augmentation or ordinal score modeling also reduces overall performance.We believe that the experiments in the revised manuscript could demonstrate the individual contribution of each component.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment

We thank the reviewers for their constructive feedback and for recognizing the potential of open-dimension image scoring (t5pe, bw2f, kQAJ, hUWn), the value of MD61 (bw2f, kQAJ, hUWn), and the relevance of evaluating held-out assessment dimensions and downstream task alignment (t5pe, bw2f, kQAJ).

Related work

We have added comparisons with PromptIQA, PPA, and open-world image aesthetic assessment (Sec. 2.3), clarifying the differences between these methods and OASIS. We have also clarified that the demonstrated setting concerns compositional generalization to unseen assessment dimensions, rather than unrestricted generalization to entirely unseen visual concepts.

Additional results

In this revision, we include:

  • Validation of synthetic annotation reliability, comparisons with teacher models, and component ablations (Tables 3, 4, and 7).
  • Additional details for MD61-Open (Appendix D).
  • Preference-reversal experiments on undegraded original images (Appendix G).
  • Downstream task performance comparisons using OASIS as a data selection criterion (Appendix H).
  • Single-domain training comparisons and training data scale ablations (Appendix I).
  • Generalization evaluations on four domains unseen during training (Appendix J).

We have also moved the case-study comparisons from the supplementary material to Appendix L of the revised manuscript, making them easier to access alongside the quantitative results.

Additional details

We have also consolidated additional implementation details, training protocols, annotation procedures, and experimental settings in the revised manuscript and appendices to improve reproducibility and readability.

Changes in the revised manuscript and appendices are highlighted in blue.

Dear Reviewers,

As the discussion period is coming to a close, I would like to encourage everyone to review the authors' responses and engage in any remaining discussions. Please indicate whether your concerns have been adequately addressed and update your scores if your assessment has changed.

Your active participation is greatly appreciated and will be valuable in reaching a fair and well-informed decision.

Thank you for your time and effort!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.