- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the NR-IQA task by freezing the vision foundation models (ViT pre-trained with different tasks) as backbone and only fit only linear probes to human quality ratings. Through experiments, the authors find that the quality-related information is most abundant in the intermediate blocks, and the retained information in the final output layer depends on the training objectives.
Critical strengths
The research topic is clear, and the results of the intermediate layer analysis are kind of enlightening.
Critical weaknesses
- The value of the contributions in the down-stream applications is not that clear. It is an interesting observation that the middle layer is superior to the output layer. However, there is a lack of a comparison with the fine-tuning counterparts. In the discussion section, the claim of obtaining quality sensitivity with existing systems through a single-layer linear layer without pre-training are relatively weak. The practical applications in the industry are not necessarily coupled with specific tasks, especially at the expense of performance. Moreover, lightweight NR-IQA can also achieve relatively competitive results.
- The experiments are not thorough enough, only discussing two datasets. In fact, there are many open-source NR-IQA datasets in this field. Authentic datasets include LIVE-Wild SPAQ, and synthetic datasets include TID2013 CSIQ, etc.
Questions for authors
- Please clearly state the motivation of the paper and the possible inspiration it may bring to the subsequent work. Specifically, how should the findings facilitate the following work or practical NR-IQA model design?
- As mentioned in Sec 3.1, the compared ViT backbones differ not only in objectives but also in the amount of training data, input resolution and training strategy. For instance, DINOv2 uses larger input resolution and CLIP employs larger training dataset. Since these factors are quite important for NR-IQA training, how can the observed findings be attributed specifically to the pre-trained objectives rather than these confounding factors?
3 comments
Thanks for the added experiments. However, the main takeaways read more like empirical design heuristics than a novel NR-IQA contribution. My rating remains unchanged.
Clarifying contributions
Thank you for reading our response and the added experiments. We agree that the paper does not propose a new NR-IQA model. It is a study of self-supervised representations, and its contribution is new knowledge about what they encode about perceived quality. Design heuristics are a consequence of that knowledge, not the contribution itself. Concretely:
-
Where quality information sits and why it is lost.
-
content–distortion separation is a genuinely different finding: the same frozen representation preserves scene identity while encoding distortion information across unseen content. Section 4.6 shows that a single frozen backbone already separates them in its intermediate blocks: it identifies which of 25 distortions was applied to an unseen scene while still representing the scene as opposed to models like Re IQA that employ separate encoders
We hope this clarifies the kind of contribution we intend. Thank you again for your time and consideration.
(Reviewer 3) Fine-Tuning Comparison, Four Suggested Benchmarks, and Design Implications
We thank the reviewer for finding the intermediate-layer analysis enlightening, and for concrete suggestions, all of which we have run.
W1 (practical value; comparison with fine-tuned models). We fine-tuned each backbone end to end on the same KonIQ-10k split (Appendix A.6.9). Fine-tuning beats the frozen output by 0.04–0.15, but a frozen linear probe on the validation-selected block comes within 0.013–0.022 of fine-tuning (e.g. CLIP 0.909 vs. 0.922), at the cost of a forward pass and a ridge fit. A per-block probe of the fine-tuned models shows why: fine-tuning mainly restores the quality information that the late blocks discard (Table 16). A small MLP on the output does not recover it (Table 11). These are single runs without hyperparameter search, which we state.
W2 (only two datasets). We added all four suggested benchmarks and AGIQA-3K, five content-disjoint splits each (Appendix A.6.8):
Authentic. Every backbone beats the baseline in all five splits on LIVE-C (four main backbones 0.699–0.779 vs. 0.620) and SPAQ (0.876–0.904 vs. 0.812). A KonIQ-10k probe transfers zero-shot to both (0.73–0.86 from the output, 0.78–0.88 from block 8). Synthetic. TID2013 and CSIQ have only 25 and 30 scenes, so pooled SROCC varies strongly between splits; within a scene, every backbone beats the baseline at the output (TID2013 0.78–0.92 vs. 0.70; CSIQ 0.84–0.97 vs. 0.83). The intermediate block beats the output for CLIP, DINOv2 and Sup on all five benchmarks, and the matched ImageNet-1k order holds on all five. Q1 (motivation and how the findings help NR-IQA design). The Introduction now names the three design choices the study informs (which backbone, which layer, how much quality-specific training), and Section 5 gives four concrete implications: (i) read a validation-selected intermediate block, which adds 0.02–0.11 SROCC at no cost and comes within 0.013–0.022 of fine-tuning; (ii) if only output features are available, prefer an objective that keeps low-level detail (among ViTs, MAE keeps 98% of its peak); (iii) a system already running one of these backbones can add quality prediction with one linear layer; (iv) keep hand-crafted statistics as a cheap, interpretable complement for diagnosing the distortion.
Q2 (confounds: data, resolution, recipe vs. objective).
Data and architecture. The matched ImageNet-1k ViT-B/16 set at 224px reaches nearly the same peak (0.866–0.883) but keeps 97.9% (MAE), 94.0% (DINO) and 88.6% (DeiT) at the output, with non-overlapping CIs (Table 8). Resolution. DINOv2 at 224px retains 85.5% vs. 88.5% at 518px on KonIQ-10k (54% vs. 48% pooled on KADID-10k), so resolution does not explain its drop. Recipe. Recipes still differ within the matched set, and the CNN families show that the architecture also matters (in ConvNeXt, reconstruction keeps less than supervision). We therefore state the objective as a major factor, not the only one, and list the recipe confound in Limitations.