acceptodds
Under review as a conference paper at ICLR 2027

DOES IMAGE QUALITY AWARENESS EMERGE FOR FREE FROM SELF-SUPERVISED PRETRAINING?

Abstract

Vision foundation models are trained to recognize and describe image content, never to judge image quality. We ask whether they nevertheless encode humanperceived quality, and where in the network. We freeze four base-size Vision Transformers (ViT-B) trained with different objectives — masked-image reconstruction (masked autoencoder, MAE), self-distillation (DINOv2), image–text contrastive learning (Contrastive Language–Image Pre-training, CLIP), and supervised classification — and fit only linear probes to human quality ratings on two image quality assessment (IQA) benchmarks, one with authentic (KonIQ-10k) and one with synthetic (KADID-10k) distortions. All four encode substantial quality information. On authentic distortions their output features outperform a probe on hand-crafted low-level statistics, reaching a Spearman rank correlation (SROCC) of up to 0.87; within a single scene they rank synthetic distortions largely in line with human ratings; and they transfer between the two kinds of distortion, where the hand-crafted baseline barely does. The information is richest in intermediate blocks, where probes reach 0.92, and how much of it survives to the output depends on the objective: pixel reconstruction preserves nearly all of it, while discriminative objectives discard part of it. Hand-crafted statistics still identify the distortion type better, and we find no reliable link between quality encoding and robustness to corruptions.

Primary area:
interpretability and explainable AI
Keywords:
representation probing, self-supervised learning, image quality assessment, linear probing, no-reference image quality assessment, vision transformers, frozen representations, emergent capabilities, transfer learning, robustness to corruptions

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QVv4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the NR-IQA task by freezing the vision foundation models (ViT pre-trained with different tasks) as backbone and only fit only linear probes to human quality ratings. Through experiments, the authors find that the quality-related information is most abundant in the intermediate blocks, and the retained information in the final output layer depends on the training objectives.

Critical strengths

The research topic is clear, and the results of the intermediate layer analysis are kind of enlightening.

Critical weaknesses
  1. The value of the contributions in the down-stream applications is not that clear. It is an interesting observation that the middle layer is superior to the output layer. However, there is a lack of a comparison with the fine-tuning counterparts. In the discussion section, the claim of obtaining quality sensitivity with existing systems through a single-layer linear layer without pre-training are relatively weak. The practical applications in the industry are not necessarily coupled with specific tasks, especially at the expense of performance. Moreover, lightweight NR-IQA can also achieve relatively competitive results.
  2. The experiments are not thorough enough, only discussing two datasets. In fact, there are many open-source NR-IQA datasets in this field. Authentic datasets include LIVE-Wild SPAQ, and synthetic datasets include TID2013 CSIQ, etc.
Questions for authors
  1. Please clearly state the motivation of the paper and the possible inspiration it may bring to the subsequent work. Specifically, how should the findings facilitate the following work or practical NR-IQA model design?
  2. As mentioned in Sec 3.1, the compared ViT backbones differ not only in objectives but also in the amount of training data, input resolution and training strategy. For instance, DINOv2 uses larger input resolution and CLIP employs larger training dataset. Since these factors are quite important for NR-IQA training, how can the observed findings be attributed specifically to the pre-trained objectives rather than these confounding factors?
3 comments
AuthorsOpenReview ↗

(Reviewer 3) Fine-Tuning Comparison, Four Suggested Benchmarks, and Design Implications

We thank the reviewer for finding the intermediate-layer analysis enlightening, and for concrete suggestions, all of which we have run.

W1 (practical value; comparison with fine-tuned models). We fine-tuned each backbone end to end on the same KonIQ-10k split (Appendix A.6.9). Fine-tuning beats the frozen output by 0.04–0.15, but a frozen linear probe on the validation-selected block comes within 0.013–0.022 of fine-tuning (e.g. CLIP 0.909 vs. 0.922), at the cost of a forward pass and a ridge fit. A per-block probe of the fine-tuned models shows why: fine-tuning mainly restores the quality information that the late blocks discard (Table 16). A small MLP on the output does not recover it (Table 11). These are single runs without hyperparameter search, which we state.

W2 (only two datasets). We added all four suggested benchmarks and AGIQA-3K, five content-disjoint splits each (Appendix A.6.8):

Authentic. Every backbone beats the baseline in all five splits on LIVE-C (four main backbones 0.699–0.779 vs. 0.620) and SPAQ (0.876–0.904 vs. 0.812). A KonIQ-10k probe transfers zero-shot to both (0.73–0.86 from the output, 0.78–0.88 from block 8). Synthetic. TID2013 and CSIQ have only 25 and 30 scenes, so pooled SROCC varies strongly between splits; within a scene, every backbone beats the baseline at the output (TID2013 0.78–0.92 vs. 0.70; CSIQ 0.84–0.97 vs. 0.83). The intermediate block beats the output for CLIP, DINOv2 and Sup on all five benchmarks, and the matched ImageNet-1k order holds on all five. Q1 (motivation and how the findings help NR-IQA design). The Introduction now names the three design choices the study informs (which backbone, which layer, how much quality-specific training), and Section 5 gives four concrete implications: (i) read a validation-selected intermediate block, which adds 0.02–0.11 SROCC at no cost and comes within 0.013–0.022 of fine-tuning; (ii) if only output features are available, prefer an objective that keeps low-level detail (among ViTs, MAE keeps 98% of its peak); (iii) a system already running one of these backbones can add quality prediction with one linear layer; (iv) keep hand-crafted statistics as a cheap, interpretable complement for diagnosing the distortion.

Q2 (confounds: data, resolution, recipe vs. objective).

Data and architecture. The matched ImageNet-1k ViT-B/16 set at 224px reaches nearly the same peak (0.866–0.883) but keeps 97.9% (MAE), 94.0% (DINO) and 88.6% (DeiT) at the output, with non-overlapping CIs (Table 8). Resolution. DINOv2 at 224px retains 85.5% vs. 88.5% at 518px on KonIQ-10k (54% vs. 48% pooled on KADID-10k), so resolution does not explain its drop. Recipe. Recipes still differ within the matched set, and the CNN families show that the architecture also matters (in ConvNeXt, reconstruction keeps less than supervision). We therefore state the objective as a major factor, not the only one, and list the recipe confound in Limitations.

Thanks for the added experiments. However, the main takeaways read more like empirical design heuristics than a novel NR-IQA contribution. My rating remains unchanged.

AuthorsOpenReview ↗

Clarifying contributions

Thank you for reading our response and the added experiments. We agree that the paper does not propose a new NR-IQA model. It is a study of self-supervised representations, and its contribution is new knowledge about what they encode about perceived quality. Design heuristics are a consequence of that knowledge, not the contribution itself. Concretely:

  1. Where quality information sits and why it is lost.

  2. content–distortion separation is a genuinely different finding: the same frozen representation preserves scene identity while encoding distortion information across unseen content. Section 4.6 shows that a single frozen backbone already separates them in its intermediate blocks: it identifies which of 25 distortions was applied to an unseen scene while still representing the scene as opposed to models like Re IQA that employ separate encoders

We hope this clarifies the kind of contribution we intend. Thank you again for your time and consideration.

Reviewer_YyCHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

It seems to be an AI-generated paper, trying to validate a well-known fact: VLM features are useful for image quality assessment (IQA).

This paper investigates whether visual representations from self-supervised and supervised pretraining implicitly capture image quality awareness. Specifically, the authors evaluate Vision Transformer (ViT) backbones pretrained under various objectives (including MAE, DINOv2, CLIP, and supervised classification) by applying linear probing on two standard Image Quality Assessment (IQA) benchmarks: KonIQ-10k and KADID-10k. The empirical findings suggest that linear probes on pretrained VLM features can achieve reasonable performance baseline levels for general IQA; however, they remain inferior to specialized hand-crafted features on targeted distortion detection tasks.

Critical strengths

None

Critical weaknesses
  1. Limited Methodological Novelty: The experimental framework is relatively straightforward and relies primarily on standard linear probing, offering limited technical contribution or new theoretical insights into feature representations.

  2. Benchmark Scope and Generalizability: The evaluation is confined to conventional IQA datasets (e.g., KonIQ-10k and KADID-10k). As a result, it remains uncertain whether these findings generalize to contemporary quality assessment paradigms, such as AI-generated content (AIGC) evaluation.

Questions for authors

The paper’s impact could be strengthened by expanding the depth of original experimental design and qualitative analysis, thereby relying less on AI auto research.

3 comments
AuthorsOpenReview ↗

(Reviewer 2) Scope of Study, New Benchmarks Including AI-Generated Images, and a New Finding

We thank the reviewer for their time and comments. Below we respond to both weaknesses with new experiments, and we also take the opportunity to clarify two aspects of the paper that we may not have described clearly enough.

Two clarifications.

The backbones. Three of the four backbones are vision-only models (MAE, DINOv2 and a supervised ViT); CLIP is the only vision–language model. The study therefore concerns general-purpose vision backbones rather than VLM features specifically. Comparison with hand-crafted features. All four backbones predict MOS better than the hand-crafted baseline, with non-overlapping confidence intervals (KonIQ-10k: 0.790–0.868 vs. 0.727), and they do so in each of five random splits. The baseline is stronger only at classifying the distortion type from the output features; intermediate blocks outperform it on that task as well (Section 4.6). W1 (limited methodological novelty). We agree that the paper does not propose a new architecture; its contribution is empirical. Instead, it provides a controlled account of where quality information resides in general-purpose backbones, how the pretraining objective shapes what reaches the output, and how to read it out, which directly informs how NR-IQA models are built on these backbones. The revision adds the following results:

Content–distortion separation without a dedicated quality encoder (Section 4.6). Re-IQA learns quality with a separate encoder and combines it with content features, and ARNIQA trains a ResNet-50 to cluster images by distortion. We test whether general backbones already separate the two factors. In intermediate blocks, a linear probe names which of 25 distortions was applied to scenes it has not seen (accuracy 0.63–0.83; chance 0.04), while the scene remains retrievable (0.96–1.00). ARNIQA's output names the distortion with accuracy 0.73, and its scene retrieval falls to 0.20 at the output. Where quality information lies and how much reaches the output. Quality information peaks in intermediate blocks. In ViTs with data, architecture and resolution held fixed, the share that reaches the output depends on the objective (Table 8), and fine-tuning on quality removes the late-block loss (Table 16). Practical consequence. A linear probe on the best frozen block comes within 0.013–0.022 SROCC of full fine-tuning (Table 15). We have also made the relation to CLIP-IQA, CONTRIQUE, Re-IQA, ARNIQA and Yadav et al. 2026 more explicit in Related Work: these works train for quality or add trainable heads, whereas we examine what general-purpose objectives encode under a frozen linear probe, at matched scale, with content controlled, and across depth.

W2 (only conventional datasets; AI-generated images). Thank you for this suggestion. We added AGIQA-3K, together with LIVE-C, SPAQ, TID2013 and CSIQ (Tables 12–14). On AGIQA-3K, every backbone outperforms the hand-crafted baseline in all five splits (the four main backbones reach 0.776–0.858 at the output, against 0.691), and for CLIP, DINOv2 and Sup the intermediate block improves on the output, as on KonIQ-10k.

Thanks for the reply. Given that the paper needs significant changes and improvements, I will keep my rating.

We thank the reviewer for reading our response and for the reply.

We would like to note that the changes are already made in the revised PDF uploaded during the discussion period, rather than planned. Regarding the two weaknesses raised in the review:

AI-generated images (W2): AGIQA-3K is now included, together with LIVE-C, SPAQ, TID2013 and CSIQ (Appendix A.6.8). On AGIQA-3K, every backbone outperforms the hand-crafted baseline in all five splits. Novelty (W1): Section 4.6 adds a new analysis showing that general-purpose backbones separate an image’s content from its distortion in intermediate blocks, without the dedicated quality encoder that Re-IQA uses or the content loss that ARNIQA incurs.

The paper also adds a matched set of ImageNet-1k backbones, two CNN families and a comparison with full fine-tuning. The main text remains within the page limit, and code for every new experiment is in the updated supplementary material.

We fully respect the reviewer’s assessment. If a specific point remains unconvincing, we would be glad to clarify it before the discussion period closes.

Reviewer_7oj3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors report an interesting behavior of one transformer architecture trained for four different goals involving vision using different training sets (or different training environments): they find that these models develop the ability to judge distortions in images in a similar way as humans do. In particular, the distances obtained from the models are highly correlated with human opinion on no-reference distortion. The authors measure this ability at different depths of the architectures. They use a linear regressor in each layer to predict the distance, and linear classifiers to identify the kind of distortion and the intensity of the distortion. The authors find that at mid-depth the models provide very good correlations with humans and that the self-supervised model (trained for different kinds of invariances) is the one that excels in correlation, which leads to the central suggestion of the work. On the other hand, the authors acknowledge moderate success in guessing the kind of distortion.

Critical strengths

S1 - Emergence of human-like perceptual properties in models trained for different vision tasks is interesting for different communities within the ICLR crowd: on the one hand it is useful to understand the behavior of the models (for those interested in machine learning) and also the behavior of humans (for theoretical neuroscientists).

S2 - The analysis done for no-reference metrics is original (similar analysis was done -see below- but only for full-reference metrics).

S3 - The per-layer analysis is interesting and results are consistent with previous literature (not cited, see examples below). The peak at mid-depth is interesting and also the fact that lower-level self supervision (the one related to recovering missing pixels, a sort of denoising) is the one that better correlates with this human opinion in the final layer.

Critical weaknesses

W1 - Missing related literature.

The authors do not cite similar analysis for full-reference metrics, namely:

Zhang et al. (2018) The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. https://arxiv.org/abs/1801.03924 (first to show the emergence of humna-like perceptual metric behavior)

Kumar et al. (2022) Do better ImageNet classifiers assess perceptual similarity better? Transactions of Machine Learning Research 2022 https://openreview.net/pdf?id=qrGKGZZvH0 (Exhaustive study of many architectures devoted to classification with different success in perceptual metrics: better performance in the goal does not imply better emergence of the metric -inverted U-curve-)

Hernández-Cámara et al. (2024) Measuring Human-CLIP Alignment at Different Abstraction Levels. ICLR Workshop on Representational Alignment 2024 https://openreview.net/forum?id=xQyhHjLGmj&noteId=GHfdrqWEj2 (First to show the emergence of perceptual distances at different abstraction levels -not only low-level distortion, but also identification of conceptual differences in images-)

Hernandez-Camara et al. (2025) Dissecting the effectiveness of deep features as metric of perceptual image quality, Neural Networks, Vol.185, doi:10.1016/j.neunet.2025.107189 (First to show per-layer analysis in the emergence of metrics and that most of them show a "peak at mid-depth". Moreover, they show that the inverted U-curve behavior also happens in many vision tasks. This means that it is not trivial to disentangle "task", "architecture" and "training environment" in the emergence of human-like distances)

W2 - The scientific question could be broader (or the research could be put in a brader context).

Measuring distances in a human-like way is not just an engineering question required in efficient image processing or a curiosity in artificial systems. It is a major question in visual neuroscience: why we perceive distances the way we do?. Classical approaches that date back to the Barlow hypothesis in the late 50s focus on the statistical properties of the environment, and reason taking into account just the computational task to be solved (regardless of architectural constraints), following a naive or strict interpretation of the separation of the Marr-Poggio levels to address vision. In this way, emergence of a perceptual behavior due to a certain task was considered a proof of the fact that this task could cause biological behavior. However, recent research has questioned such naive interpretation because the Marr-Poggio levels are not independent and one cannot tell about the relevance of tasks without taking into account the architecture (See Poggio 2021, and the references below. Similarly to Hernandez et al. Neur.Nets 25 cited above, Li et al. 22 and Malo & Hernandez 24 talk about the inverted U-curve behavior -first shown by Kumar 22-).

T. Poggio (2021) From Marr’s vision to the problem of human intelligence CBMM Memo 118, 2021

Li et al. (2022) Contrast sensitivity functions in autoencoders J. Vision, 22(6), 2022

Pillow (2024) Cross Talk opposing view: Marr's three levels of analysis are not useful as a framework for neuroscience. J. Physiol., 602 (9) (2024), p. 1919 https://doi.org/10.1113/JP279550

Malo & Hernandez-Camara (2024) A separate theory-on-top level may be inspiring, but it is neither separate nor enough J. Physiol., 602 (9) (2024), p. 1919 https://physoc.onlinelibrary.wiley.com/action/downloadSupplement?doi=10.1113%2FJP279550&file=tjp16021-sup-0001-Comments.pdf

W3 - Experiments are scarce to fully suport the suggestion that it is self-supervision what it takes to solve that problem.

According to the non-trivial dependence between task and architecture mentioned above (refs. W1 and W2) more experiments are needed to suggest that self-supervision is the reason for the emergence of the human-like behavior. I certainly "believe" that non supervised (as in your DINO case) has to be the reason for biological behavior, but note that in your "limited" experiments the other non-supervised task (MAE) does not reproduce human behavior better than suppervised tasks... The question requires more exhaustive experiments (broader range of architectures...).

W4 - No major conclusion/insight is extracted from the fact that the peak is in the middle region of the models.

Incidentally this was also the problem in Hernandez et al. 25. Again, the broader question is not easy to answer and it is still open.

** I am happy to raise my score if the authors include the suggested discussions

Questions for authors

1.- Please consider citing/discussing the references in W1

2.- Please consider citing/discussing the question/refs in W2

3.- Please acknowledge W3/W4 as limitations

1 comment
AuthorsOpenReview ↗

(Reviewer 1) Broader Architectures, References, and What the Mid-Network Peak Encodes

We thank the reviewer for the detailed review, the references, and the offer to raise the score if the discussion is extended. We have added all of it, and the new experiments answer W3 and W4 directly.

Before turning to the individual points, we would like to clarify three aspects of the paper that we did not explain clearly enough in the submission, as they bear on W3:

Linear probes rather than feature distances. Our setting is no-reference: we fit a linear probe that predicts each image's MOS from that image's frozen features alone. The works the reviewer kindly pointed us to in W1 study the complementary full-reference setting, measuring distances between a distorted image and its reference. We agree this distinction deserves to be explicit and now discuss it in Related Work ("Deep features as perceptual metrics"). The role of self-supervision. We share the reviewer's caution here, and we did not intend to claim that self-supervision is the cause. The supervised classifier also encodes quality (Section 4.1: "the effect is not specific to self-supervision"). The title is meant as a question, and the supervised control, together with the new CNN results, suggests that quality awareness emerges under supervised and self-supervised objectives alike. MAE compared with the supervised model. We may not have presented this comparison clearly. At the output, MAE predicts MOS better than the supervised model on KonIQ-10k (0.868 vs. 0.790, non-overlapping CIs), while at the best intermediate block the two are similar (0.887 vs. 0.900). The difference therefore lies mainly in how much of the peak survives to the output (97.9% vs. 87.8%). W1 (missing references). Added: Zhang et al. 2018 (LPIPS), Kumar et al. 2022, Hernández-Cámara et al. 2024 and 2025, in a new Related Work paragraph, and in Section 4.4, where the mid-network peak matches Hernández-Cámara et al. 2025 and our CNN result echoes Kumar et al. 2022.

W2 (neuroscience framing). Added a Discussion paragraph, "Relation to biological vision, and what the mid-network peak implies", citing Barlow 1961, Marr 1982, Poggio 2012, Li et al. 2022, Lengyel 2024 and Pillow 2024. We place the mid-network peak at the algorithmic level (it appears under every objective with architecture fixed) and the objective-dependent loss at the computational level, and state that settling the latter needs models trained identically except for the objective.

W3 (too few experiments to attribute the effect to the objective; broader architectures). We ran the experiments the reviewer asked for (Appendix A.6.1–A.6.2):

Matched ImageNet-1k ViT-B/16 set, 224px. All three reach almost the same KonIQ-10k peak (0.866–0.883) but keep different shares at the output: MAE 97.9%, DINO 94.0%, DeiT 88.6%, with non-overlapping output CIs. The same order holds within KADID-10k scenes and at the output on all five further benchmarks. Resolution. DINOv2 at 224px loses as much as at 518px (85.5% vs. 88.5% retained on KonIQ-10k), and DINO keeps far more than DINOv2, so DINOv2's drop is specific to its recipe, not to self-distillation. CNNs. Every ResNet-50 and ConvNeXt-B peaks before its output and keeps 83–94% at the output, so the peak and the loss are not specific to transformers. But the ordering changes: in ConvNeXt, reconstruction keeps less than supervision (83.4% vs. 86.8%). This supports the reviewer's point that task and architecture interact, and we now say so in Section 4.4, citing Kumar et al. 2022 and Hernández-Cámara et al. 2025. We revised the claim to "the objective is a major factor, not the only one", and list in Limitations that recipes still differ. W4 (no insight from the mid-depth peak). We tested what the peak contains and why it fades:

It holds on synthetic data (KADID-10k peaks at blocks 4–7; DINOv2 pooled 0.837 → 0.402 at the output, Table 10). The output loss is real. An MLP on the output never beats the linear probe on KonIQ-10k (Table 11), and fine-tuning on quality removes the loss: the fine-tuned models' probe rises to the last block (DINOv2 0.918, MAE 0.905; Table 16). This points to the pretraining objective, not the architecture, as the source of the late-block loss. What the peak encodes (new Section 4.6). The intermediate blocks hold a content-independent distortion code: they name the distortion of unseen scenes while the scene stays recognizable. This code peaks in the same block as the KADID-10k quality probe for CLIP, DINOv2 and Sup, and fades towards the output most for DINOv2 (0.83 → 0.33) and least for MAE (0.72 → 0.57), mirroring the quality loss. What it is not. Distortions resembling DINOv2's training augmentations are not suppressed more than others (p = 0.24, A.6.6). Late blocks do become less sensitive to corruptions within each discriminative backbone (A.6.5), but this does not explain differences between backbones.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You to All Reviewers: New Evidence Across Backbones, Benchmarks, and Fine-Tuning

We thank the reviewers for their careful reading. The revision runs every experiment the reviewers asked for, plus several controls, and changes the paper as follows (new material is in Sections 3.1, 4.4, 4.6 and Appendix A.6):

Broader backbones. A matched set of ImageNet-1k ViT-B/16 checkpoints that differ mainly in objective (MAE, DINO, DeiT), DINOv2 at 224px to isolate resolution, and two CNN families (ResNet-50: supervised, DINO, MoCo v3; ConvNeXt-B: supervised vs. ConvNeXt V2 masked reconstruction). Tables 8–9. Five more benchmarks. LIVE-C, SPAQ, AI-generated AGIQA-3K, TID2013 and CSIQ, five content-disjoint splits each, plus zero-shot transfer from KonIQ-10k. Tables 12–14. Full fine-tuning. End-to-end fine-tuning of each backbone on KonIQ-10k, and a per-block probe of the fine-tuned models. Tables 15–16. Controls on the mid-network peak. A KADID-10k layer sweep, an MLP probe, a per-block invariance curve, an augmentation test, and the number of training scenes. Appendix A.6.3–A.6.7. A new result: content–distortion separation (Section 4.6). On the three benchmarks where every scene appears with every distortion, a linear probe on intermediate blocks names which of 25 distortions was applied to scenes it never saw (best block 0.63–0.83 for ViTs, 0.56–0.73 for CNNs; chance 0.04; hand-crafted statistics 0.51; untrained networks 0.17–0.29), while the scene is still retrieved at 0.96–1.00 top-1. ARNIQA, a ResNet-50 trained specifically for this, names the distortion no better (0.73 at its output on KADID-10k) and loses the scene (retrieval 0.87 mid-network → 0.20 at the output). A single general-purpose representation thus holds both factors, without the separate quality encoder that Re-IQA uses. Claims narrowed where the new data require it. The objective is "a major factor, not the only one": in ViTs reconstruction keeps the most quality information at the output, but in ConvNeXt it keeps less than supervision (83.4% vs. 86.8%). Fine-tuning on quality removes the late-block loss.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.