acceptodds
Under review as a conference paper at ICLR 2027

Is One Direction Enough? Visual Embeddings Rankability for Image Quality Assessment

Abstract

Image quality assessment (IQA) models, from classic deep neural networks to multimodal large language models (MLLMs), are evaluated by how well their predicted scores correlate with human mean opinion scores (MOS). However, this evaluation does not reveal that the hidden representations are rankable by quality scores. In this work, we study representations derived from hidden layers of IQA models and propose to estimate a single linear direction, the rank-axis, and measure whether projections onto it order images by quality scores (rankability). Additionally, we propose the Rank-Axis Isometry Score (RAIS) to quantify how well their spacing reflects the size of quality differences. To test whether the embedding space itself is ordered by quality scores, we introduce three quality-embedding alignment measures computed on distances in the native feature space. Across nine IQA models, three layer depths and eight benchmarks, one rank-axis reaches a rankability of at least on six datasets, and at least one of the rank-axes across hidden representations beats the model's own fine-tuned head in 52 of 70 encoder-dataset pairs. Spacing along the axis follows the order (Spearman between RAIS and rankability), whereas the embeddings in the original latent space are only weakly ordered by quality (mean Anchor-QEA ). Rank-axes estimated on different datasets are nearly orthogonal, yet keep 80% of their rankability on unseen datasets; their agreement under the data covariance predicts this transfer (Spearman ). Our findings suggest that a single linear direction captures much of the perceptual quality ordering in visual embeddings, even when it is not reflected in their dominant geometry.

Primary area:
interpretability and explainable AI
Keywords:
Image Quality Assessment, Representation Learning, Visual Embeddings, Rankability

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_BPXUOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors investigate whether a single linear direction, a rank-axis, in the frozen latent feature space of an IQA model can order images by perceptual quality. They introduce the Rank-Axis Isometry Score (RAIS) to measure whether distances along the axis reflect quality differences, and three measures, Anchor-QEA, ordering rate, Global-QEA, to assess whether the embedding space itself is organized by quality. Their empirical studies find that a single rank-axis often beats the model's fine-tuned head especially at earlier layers, and the embedding space is only weakly ordered by quality. They also show that rank-axes transfer across datasets are well-predicted by covariance-weighted agreement, and local geometry correlates with cross-dataset agreement. Their conclusion is that a single direction is sufficient to encode perceptual quality, but the underlying embedding geometry is not.

Critical strengths
  1. Introduce a novel perspective of IQA through the lens of rankability and embedding geometry in latent feature representations.
  2. Propose several novel diagnostic tools, RAIS, Anchor-QEA, ordering rate, Global-QEA, to measure whether distances along the axis reflect quality differences and whether the embedding space itself is organized by quality.
  3. Have some interesting empirical findings: a simple ridge direction outperforms fine-tuned heads in many cases under the current test conditions; early layers can match last layers; covariance-weighted agreement well explains transfer and outperforms raw angle.
Critical weaknesses
  1. Despite the paper’s appreciated contributions and interesting observations, its central claim, that a single linear direction is sufficient for quality assessment, remains questionable. It appears surprisingly strong and may be an artifact of the limitations of current IQA benchmarks, which predominantly contain a narrow set of synthetic distortions (e.g., blur, noise, compression) that could induce a simple, one-dimensional quality manifold. By contrast, more sophisticated perceptual experiments often reveal human quality perception to be multidimensional, and subjective judgments are frequently influenced by experimental setup or instructions. Consequently, the conclusion that quality is “linearly separable” may not generalize to more diverse or authentic distortions, or to a broader range of content types.
  2. The current study focuses on IQA-specific models. It would be interesting to see whether general-purpose vision encoders also exhibit rankability for quality, or whether this is specific to models trained for IQA.
  3. The results of the paper seem to suggest that the fine-tuned heads are nearly useless, other than overfitting the data. But if the heads were trained on their authors' splits, then the comparison is not completely fair.
Questions for authors
  1. Your title, “Is One Direction Enough?”, is bold and provocative, but concluding that one direction is sufficient may be too strong, as it could be an artifact of the limitations of current IQA benchmarks. That said, the paper’s real contributions (analyzing latent space structure, introducing novel diagnostic tools, and examining cross-dataset transfer) are significant. Would you consider a major rewrite that frames the findings more cautiously, adjusts this central claim, and discusses the conditions under which a single direction may or may not suffice?
  2. How does the rankability of IQA-specific models compare to that of generic vision encoders? Would a model not trained for IQA (e.g., DINOv2) also exhibit a strong rank-axis?
  3. The paper reinforces the idea that deep latent feature spaces hold the key to quality assessment. Similar insights motivated prior IQA work such as LPIPS and deepSSIM, but those methods rely on distance-based or structural similarity comparisons in feature space. In contrast, this paper advocates for a single linear direction. These two views appear to conflict. It would be valuable to add analysis and discussions.
4 comments
AuthorsOpenReview ↗

W1 / Q1 - Is one direction enough, or an artifact of the benchmarks?

Thank you for this question. We intended to frame the title as a question; however, we updated the paper for clarity.

First, the result is not limited to synthetic distortions. Besides LIVE, TID2013 and KADID-10k, our eight datasets include restoration and GAN outputs (PIPAL), images with authentic distortions (LIVEC, KonIQ-10k, SPAQ) and AI-generated images (AGIQA-3K). The table gives the rankability of the rank-axis at ℓ_last (mean over the 9 IQA encoders) and how much a two-layer MLP on the same features adds:

IQA dataset categoryDatasetsR_ρ (ℓ_last)MLP − rank-axis
SyntheticLIVE / TID2013 / KADID-10k0.956 / 0.769 / 0.885+0.008 / +0.028 / +0.003
Restoration / GANPIPAL0.470+0.060
AuthenticLIVEC / KonIQ-10k / SPAQ0.852 / 0.904 / 0.916−0.014 / −0.001 / −0.001
AI-generatedAGIQA-3K0.841−0.005

On authentic and AI-generated images, one direction ranks images about as well as on synthetic ones. This concludes that the single direction performance gains are not an artifact of simple synthetic distortion.

Second, there are cases where a single direction may not fully capture the diversity of IQA latent space. Its relative performance is weakest on TID2013, which encompasses 24 distortion types, and PIPAL, whose test set includes distortion types unseen during training. These are also the datasets where the MLP provides the largest improvements. However, the rank-axis still remains competitive with the MLP, achieving an SRCC within 0.06 of the MLP, demonstrating that the one-dimensional projection captures most of what a nonlinear model can extract from the same features, even under substantial distortion diversity and distribution shift.

Third, we acknowledge that human quality perception is inherently multidimensional and may vary depending on the subjective evaluation protocol. The eight standard IQA benchmarks considered in this work provide a single MOS for each image, thereby allowing us to examine whether this scalar quality score can be captured along a single feature-space direction. Our conclusions are therefore scoped to this evaluation setting. Within this scope, RAIS and Anchor-QEA further extend the analysis beyond ranking consistency by examining spacing and local-ordering in native embedding space. Quality-specific textual descriptions and judgements collected under different subjective study protocols are left for future work as mentioned in conclusion.

AuthorsOpenReview ↗

W2 / Q2 - Generic encoders.

We added two generic encoders: CLIP ViT-B/16 and DINOv2 ViT-B/14. We extracted their features at the same three layer depths and evaluated them with the same protocol on all eight datasets. The table compares them with the mean of the 9 IQA encoders. All values are means over the eight datasets, weighted by the number of test images.

R_ρ ℓ_earlyR_ρ ℓ_midR_ρ ℓ_lastRAIS ℓ_lastAnchor-QEA ℓ_last
Mean (9 IQA encoders)0.8140.8250.8410.6660.453
CLIP ViT-B/160.8100.7660.7790.5350.092
DINOv2 ViT-B/140.8070.7400.6260.3810.050

Analysis. At ℓ_early, the generic encoders are closely as rankable as the IQA encoders. CLIP reaches R_ρ = 0.810, and DINOv2 reaches 0.807, against 0.814 for the IQA mean.

The two groups separate with depth. The rankability of the IQA encoders increases from 0.814 at ℓ_early to 0.841 at ℓ_last. The rankability of CLIP decreases to 0.779, and that of DINOv2 decreases to 0.626.

The native embedding space of the generic encoders is not ordered by quality. Their Anchor-QEA stays at or below 0.13 at every layer, while the IQA mean reaches 0.453 at ℓ_last. The revised paper reports these results in §5.1 and Appendix A.4 (Tables 6–7).

AuthorsOpenReview ↗

W3 - Fairness of the head comparison.

Please refer to our answer to Reviewer DKch (Q3) and to Table 11 of the revised paper for the full comparison.

The rank-axis stays close to the fine-tuned heads and beats them in 52 of the 70 encoder-dataset pairs. Against heads trained on the target dataset, it is better in 13 of 26 pairs, and the head's advantage in the other 13 pairs is small (mean margin −0.043).

AuthorsOpenReview ↗

Q3 - LPIPS / DeepSSIM vs a rank-axis

LPIPS and DeepSSIM are full-reference approaches, but they address different objectives. While LPIPS[1] measures perceptual similarity, DeepSSIM[2] focuses on structural similarity. Our work instead investigates how perceptual quality information is organized in pretrained visual embeddings. Therefore, we replaced DeepSSIM with DISTS[3] and DeepDC[4] in our quantitative comparison below. To establish an empirical connection with perceptual similarity measures, we compare our rank-axis applied to reference-distorted feature displacements and established FR metrics on the same image pairs.

SRCCLIVETID2013KADID-10k
Rank-axis0.9600.8410.897
LPIPS (AlexNet)0.9140.7310.823
DISTS0.9550.7210.819
DeepDC0.9450.7060.834

This table is added to the revised paper as Appendix A.11 (Table 29).

[1] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on computer vision and pattern recognition, pp. 586–595. IEEE, 2018.

[2] Keke Zhang, Weiling Chen, Tiesong Zhao, and Zhou Wang. Structural similarity in deep features: Unified image quality assessment robust to geometrically disparate reference. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3):2581–2595, 2026.

[3] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2567–2581, 2022.

[4] Hanwei Zhu, Baoliang Chen, Lingyu Zhu, Shiqi Wang, and Weisi Lin. DeepDC: Deep distance correlation as a perceptual image quality evaluator. IEEE Transactions on Image Processing, 34: 7859–7873, 2025.

Reviewer_UutJOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work asks whether frozen representations of popular IQA models order images by perceptual quality. While an existing work shows that vision embeddings usually contain “rank axes” ordering images by quality, this work aims to pose three separate questions: (i) whether rank axes also capture the magnitude of quality score differences, (ii) whether quality structure beyond a single direction exists in the embedding space, and (iii) whether the quality rank axis transfers across datasets. Empirical investigation (comparing rank axes with the finetuned heads of the given models) is performed using features from different layers of 9 IQA models, on 8 benchmarks. The paper organizes the properties under study into a gradation: order, spacing, geometry and transfer. The main claims are: (i) visual embeddings are rankable by quality, often beating the finetuned heads themselves, (ii) Spacing along rank axis usually reflects magnitude differences in quality but similar quality images do not usually lie close in the embedding space, and (iii) Rank axes retain the majority of their rankabilities upon transfer to a different dataset.

Critical strengths

Some observations are potentially interesting.

Much of the analysis in the submission looks like a special case of the more general study [1]. Nevertheless, some observations are interesting in isolation: (i) Rank axis transfer follows content type (§5.2), (ii) Spacing follows order (§5.3). But these observations do not support a substantial, original central claim that is sufficiently motivated in the paper. Therefore, they do not significantly change my final assessment of the paper.

Critical weaknesses

Limited originality/significance of the research question and findings.

I fear that the paper does not sufficiently explain what it adds beyond the original rankability paper [1], and what one may learn from its findings.

  • The first finding, §5.1: visual embeddings exhibit ordering along the rank axis, was already shown in the original rankability work [1], for attributes beyond (and including) quality, showing that linear regression over frozen features often beats dedicated models. The paper does not establish why an extended investigation into quality alone, arriving at essentially the same conclusion as prior work, is warranted.

  • Likewise, the original rankability work [1] also explores transfer among datasets (see Table 4 in [1]). This paper’s additional observation (rank axis transfer follows content type, §5.2) is potentially interesting, but in my opinion, not sufficient to carry the motivation by itself. The paper also does not substantiate what “content type” refers to concretely, and how generalizable this observation is; in particular, all datasets seem to somewhat transfer to all datasets in general, which does not add significant information beyond the known results of [1]. The other observation that transfer is asymmetric can, similarly, also be found in [1].

  • “The full space is only weakly quality-ordered” in §5.3 essentially means that on average, images of similar quality are not close together in the original full-dimensionality feature space. I do not yet see a reason to expect otherwise, and how this knowledge would matter in practice; I would expect considerably more justification/motivation toward the significance of this observation as a main contribution.

  • Finally, regarding cΣc_{\Sigma}, which is the pearson correlation between the projected scores on two rank axes, the finding in §5.4 is acknowledged in the paper itself as being “partly expected by construction” (lines 281-282). “Product model explains transfer” (line 463) is also largely an empirical verification of Eqn. (10). Critically, knowing this relationship does not help us predict whether the rank axis from one dataset will transfer to another dataset: for computing the two quantities in the right hand side of Eqn (10), we would have to obtain the rank axes of both datasets, at which point we might as well use the rank axis of the target dataset itself.

Questions for authors

Please treat Weaknesses as questions; my main concern is that the paper does not add sufficient insight beyond existing works, this is divided up into concrete points above so that they can be hopefully addressed separately during the rebuttal.

In addition,

Q1. My impression is that the current manuscript lacks a core claim/takeaway. I may have misunderstood this: in this case, could the authors clarify the central contribution (beyond the known empirical finding of vision embeddings being rankable)?

Q2. Could the authors substantiate the practical motivation for measuring (i) RAIS and (ii) closeness in embedding space (the three QEA metrics)? As of now, it is unclear to me why these metrics are useful for quality assessment, and why one might need three QEA metrics rather than any one of them.

[1] Sonthalia et al., “On the Rankability of Visual Embeddings.” NeurIPS 2025.

7 comments
AuthorsOpenReview ↗

W1 / Q1 - What does the paper add beyond [1]?

We agree that [1] shows that visual embeddings are rankable, however as stated in the introduction, we study three questions that the rankability framework of [1] does not address:

  1. Does the rank-axis capture the magnitude of quality differences, not just their order? Rankability considers only projection order.
  2. Does a quality structure exist beyond a single direction, such that similarly rated images are also close in the embedding?
  3. Is the quality rank-axis transferable across datasets, and what explains it?
PropertySonthalia [1]Our workReference in PDF
Order along one direction✓ (several attributes; only 1 image quality dataset, generic encoders)✓, 9 IQA models, 8 IQA benchmarks, 3 layer depths§3.2; §5.1, Fig. 2; App. A.2, Tables 3–4
Spacing: do distances along the axis reflect the size of quality differences?✗RAIS, new measure with Proposition 1§3.3, App. B; §5.2, Table 1
Native latent space: are images of similar quality close together?✗Anchor-QEA, ordering rate, Global-QEA§3.4; §5.2, Table 1; App. A.9, Tables 22–28
Why nearly orthogonal axes still transfer across datasets✗Covariance-weighted agreement c_Σ, compared with the raw angle§3.5; §5.4, Fig. 3; App. A.8.4, Table 20
Layer depth and inference cost✗An early layer is often enough, at a fraction of the cost§5.1, Fig. 2; App. A.6, Table 10
IQA vs generic encoders✗Generic encoders lose the quality direction with depth§5.1; App. A.4, Tables 6–7

Neither [1] nor existing IQA metrics, which report only SRCC/PLCC of the head output, measure magnitude or latent-space alignment; we needed measures for both. The core takeaway: in IQA representations, quality is ordered along one direction with faithful spacing, but not in the geometry of the native space, and this organisation differs between models and layers.

AuthorsOpenReview ↗

W2 - "Content type" and transfer

By content type, we meant the category of the IQA dataset, defined by how its images and distortions are produced: synthetic distortions (LIVE, TID2013, KADID-10k), outputs of GAN-based restoration and compression (PIPAL), authentic captures (LIVEC, KonIQ-10k, SPAQ) and AI-generated images (AGIQA-3K). For clarity, we replace "content type" with "IQA dataset category" in the revised paper. To measure how much of its rankability an axis keeps on another dataset, we use the retention R_ρ(a→b) / R_ρ(b→b), computed over the 56 ordered dataset pairs.

Mean retention by IQA dataset category (rows: dataset the axis is fit on; columns: dataset it is tested on):

Fit on ↓ / Tested on →SyntheticPIPALAuthenticAI-generated
Synthetic0.910.790.710.71
PIPAL0.69–0.570.57
Authentic0.801.050.930.80
AI-generated0.800.960.75–

Synthetic: LIVE, TID2013, KADID-10k; authentic: LIVEC, KonIQ-10k, SPAQ. "–": the category has only one dataset.

Analysis. All datasets transfer to some degree, consistent with [1]; what we add is how much transfers between IQA dataset categories. Axes keep most of their rankability within the same category (0.91 for synthetic, 0.93 for authentic) and less across categories (0.57–0.80), except when PIPAL is the target, whose own axis is weak. Axes fit on PIPAL transfer the least to every other category. This table is added to the revised paper as Appendix A.8.5 (Table 21), and a reference from §5.3.

AuthorsOpenReview ↗

W3 - Latent-space ordering

"images of similar quality are not close together in the original full-dimensionality feature space. I do not yet see a reason to expect otherwise"

We see three concrete reasons to expect otherwise for IQA models:

  1. Training target. IQA models are trained to map images with the same MOS to the same output; features of images with similar MOS are therefore expected to lie close together.
  2. Explicit training objectives. Some IQA models are trained to place images of similar quality close together. ARNIQA learns a distortion manifold in which images with the same distortion type and level are close regardless of content, and QualiCLIP is trained for consistent representations across content at the same severity level.
  3. Common practice. Distance-based IQA (LPIPS, DISTS) and nearest-neighbour or retrieval use of the features assume that feature distance reflects quality.

Anchor-QEA tests this expectation directly. It holds only partly, and differently for each model and layer, while rankability stays similar (in-dataset; mean over the 8 datasets weighted by the number of test images, as in Tables 6–7; per-dataset values for all layers in Appendix A.9, Tables 22–28):

Encoderℓ_early R_ρℓ_early Anchor-QEAℓ_mid R_ρℓ_mid Anchor-QEAℓ_last R_ρℓ_last Anchor-QEA
QualiCLIP0.7760.1320.7960.1600.8110.147
ARNIQA0.7580.1630.8040.1960.8210.271
LIQE0.7970.1620.8200.1300.8650.405
FGResQ0.8040.1380.7660.1160.8230.412
Q-Insight0.8240.1440.8530.5290.8510.412
Q-ReAlign0.8260.5170.8110.5300.8170.536
MANIQA0.8380.4090.8440.5030.8460.581
Q-Align0.8740.5740.8710.6370.8710.664
VisualQuality-R10.8260.1350.8600.6410.8630.647
Mean (9 IQA encoders)0.8140.2640.8250.3820.8410.453
CLIP ViT-B/16 (generic)0.8100.1280.7660.0620.7790.092
DINOv2 ViT-B/14 (generic)0.8070.1320.7400.0930.6260.050

Analysis.

  • The two models trained to bring similar-quality images together have the least quality-ordered space at ℓ_last (QualiCLIP 0.147, ARNIQA 0.271), while their rankability is similar to the other models, and their Anchor-QEA stays at most 0.27 at every layer. Their training objective therefore does not produce the intended geometry.
  • Models with similar rankability differ by a factor of more than four in Anchor-QEA (at ℓ_last, 0.15–0.66 at R_ρ 0.81–0.87). SRCC of the predicted score cannot reveal this difference.
  • IQA training does organise the space partly: Anchor-QEA of the IQA encoders rises from 0.26 at ℓ_early to 0.38 at ℓ_mid and 0.45 at ℓ_last, whereas that of the generic encoders stays at or below 0.13 at every layer. At ℓ_early, six of the nine IQA encoders (0.13–0.16) are close to the generic encoders; the ordering appears only in deeper layers.
Reviewer_DKchOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the rankability of image quality assessment (IQA) models. It introduces a rank axis for ordering images by their predicted quality scores and proposes RAIS to quantify how differences in image quality affect the RAIS space. The paper presents several empirical findings and evaluates its approach across a range of IQA models and benchmarks.

Critical strengths
  1. The paper analyzes whether embeddings from existing IQA models can be projected onto a rank axis to order images by quality. It examines embeddings from early, middle, and late layers.
  2. It presents empirical evidence that the proposed axis can capture both image rankings and differences in image quality.
  3. It evaluates the approach across a range of IQA models and benchmarks using several evaluation criteria.
Critical weaknesses
  1. Although the paper provides evidence that image rankings can be captured by projecting embeddings onto certain directions, this finding may be expected because IQA models are trained to predict image quality scores. The paper should clarify the novelty and practical significance of this projection. In particular, it is unclear how the proposed analysis can be used beyond characterizing existing models, given that it does not affect how those models are trained or learned. The authors should explain what new insights or practical applications the projection enables.

  2. The paper presents a broad range of empirical results and metrics for evaluating rankings, quality differences, and other properties. However, the central message is difficult to identify. The authors should focus on the most compelling observations, analyze them in greater depth, and explain their significance and implications.

Questions for authors
  1. Some figures are not sufficiently explained in the paper. Figure 1, for example, is not discussed in the Introduction, and its connection to the paper’s main findings is unclear. In particular, images A and C have similar MOS values but appear far apart in Fig. 1(b). The authors should explain this discrepancy and clarify what the figure is intended to illustrate.

  2. Figure 2 shows that PIPAL exhibits a markedly different pattern from the other benchmarks. Does this indicate that the reported observations do not hold consistently across datasets? The authors should discuss this difference and its implications for the generality of their findings.

  3. Lines 356–357 state that the rank axis outperforms the model’s own prediction head in more than half of the cases. Does this mean that it underperforms the head in the remaining cases? The authors should report the results more precisely and discuss whether this level of performance supports the claimed effectiveness of the rank axis.

  4. The paper includes many ablation studies, but their presentation makes the main observations less clear. The authors should streamline or reorganize the ablations to better foreground the paper’s central findings.

7 comments
AuthorsOpenReview ↗

W1 - Is rankability expected? Novelty and practical significance

Our work examines how IQA models' hidden representations are organized, we analyse these hidden representation across network layers using complementary measures of quality ordering (rankability), spacing (RAIS), and geometric alignment (QEA).

While existing IQA models, including MLLM-based approaches, are primarily evaluated through prediction metrics such as SRCC and PLCC against MOS, these metrics do not reveal how quality information is encoded within their representations or at which layers. Our work complements conventional IQA evaluation by providing a representation-level analysis of where and how quality information is encoded.

PropertyQuestionMeasureWhat SRCC/PLCC of the predicted score cannot show
OrderDoes one direction in the features order images by quality?R_ρWhether quality is encoded in the representation itself, not only produced by the prediction head
SpacingDo distances along this direction grow with quality differences?RAISWhether score differences are meaningful, not only their order ([1] measures order only)
GeometryAre images of similar quality close in the native feature space?Anchor-QEAWhether feature distances can be used directly
DepthWhich layer is sufficient to extract the quality score, and which layer has the representation best structured by quality?R_ρ, RAIS and Anchor-QEA per layerWhich layer to use and how its representation is organised; SRCC/PLCC is measured only at the output

Rankability across depth differs by model family. Mean in-dataset R_ρ over the encoders of each family and the eight datasets:

Familyℓ_earlyℓ_midℓ_last
CNN (ARNIQA, MANIQA)0.7610.7970.806
CLIP-based (LIQE, QualiCLIP, FGResQ)0.7650.7800.829
MLLM (Q-Align, Q-ReAlign, Q-Insight, VisualQuality-R1)0.8220.8290.830

Rankability increases with depth in all three families, but by different amounts: by 0.045 and 0.064 from ℓ_early to ℓ_last for CNN and CLIP-based encoders, and by only 0.008 for MLLMs, which already reach R_ρ = 0.822 at ℓ_early. An early layer can therefore be chosen for fast quality prediction; for Q-Insight and VisualQuality-R1, whose ℓ_early lies in the vision tower, this needs only about 10% of the FLOPs (below).

Our measures reveal what IQA training adds across layers. Generic encoders never trained for quality are as rankable as IQA encoders at ℓ_early (CLIP 0.810, DINOv2 0.807 vs IQA Encoder mean 0.814) but lose it with depth (0.779, 0.626 vs 0.841 at ℓ_last). Their native space is also less ordered by quality: Anchor-QEA of the generic encoders is 0.13 at ℓ_early and falls to 0.05–0.09 at ℓ_last, while that of the IQA encoders rises from 0.26 to 0.45. These layer-specific differences are identified by our measures and are not visible in the correlation of the predicted score with MOS, which is computed only at the output.

Practical use: selecting a layer for fast inference. Rankability identifies the earliest layer whose features are already ordered by quality. The network can then be run only up to that layer, and the quality score is a single inner product of the features with the rank-axis, with no head and no text generation. At ℓ_early this needs only a fraction of the FLOPs of the full scoring pass, while R_ρ stays above the fine-tuned head or within 0.03 of it:

ModelFLOPs at ℓ_early (% of full pass)Speed-upR_ρ at ℓ_earlyFine-tuned head
Q-Insight9.9%9.3×0.8240.799
VisualQuality-R111.3%7.4×0.8260.854
FGResQ21.0%3.6×0.8040.605

(Refer to Appendix A.6, Table 10, for all nine encoders.)

AuthorsOpenReview ↗

W2 / Q4 - Central message and manuscript update for clarity.

We updated the manuscript to highlight the central message that quality in IQA representations lies along one direction rather than in the geometry of the native feature space, and that this organisation differs between models and layers. In points:

  1. One direction orders images by quality. On average over the nine IQA models, the rank-axis reaches R_ρ = 0.841 at ℓ_last, and a nonlinear model on the same features adds little (Appendix A.3, Table 5).
  2. The direction also keeps the size of quality differences. RAIS measures this spacing, which order alone does not determine (Proposition 1).
  3. The native space is only partly ordered by quality. Images of similar quality are often not close together, and models with similar rankability differ strongly in Anchor-QEA (see UutJ W3). Distances in the full feature space therefore mix quality with other changes: in the full-reference setting, the rank-axis applied to the reference-distorted displacement predicts quality better than LPIPS, DISTS and DeepDC (Appendix A.11, Table 29).
  4. IQA training builds the direction with depth. Generic encoders are as rankable as IQA encoders at ℓ_early but lose it in deeper layers. An early layer is often enough, at a fraction of the cost (Appendix A.6, Table 10).
  5. Nearly orthogonal axes still transfer because they agree in the directions in which the images vary; c_Σ explains this where the raw angle fails (§5.4, Table 20).

Each measure answers one of these questions: R_ρ for order, RAIS for spacing, Anchor-QEA for the native geometry and c_Σ for transfer. We have updated the manuscript for clarity:

  • The introduction now states this central message.
  • The start of §3 now says why each measure is needed when the others exist ( Table in W1)
  • The ablations and the appendix have been restructured for better clarity.
AuthorsOpenReview ↗

Q1 - Figure 1, images A and C

Figure 1 is introduced in the introduction ("Figure 1 illustrates how the learned rank-axis can reorganize the embedding…"). In the revised paper, we have updated Figure 1 (labelled axes and a strip showing the projection onto the rank-axis alone), its caption (test SRCC of both views) and the introduction (the pair distances below). The figure contrasts the initial view (two leading principal components) with the rank-axis view, whose horizontal coordinate is the projection onto the rank-axis:

Test SRCC with MOS (LIQE, ℓ_last, LIVEC, 230 test images)
Initial view (PC1)0.875
Rank-axis view (rank-axis)0.938

We thank the reviewer for pointing out the pair A–C. In the initial view, the pair A–B (MOS gap 44.3) lies closer than the pair C–D (MOS gap 1.9) (1.66 vs 3.00). In the rank-axis view this reverses: A–B move apart (3.10) and C–D come together (0.37). A and C have a moderate MOS gap (7.6), and they are close along the rank-axis (0.44); their visible separation in panel (b) is along the vertical axis, which is not a quality direction.

Distances between the pairs in the two views of Figure 1:

PairMOS gap2D distance in initial view2D distance in rank-axis viewDistance along rank-axis
A–B44.31.663.102.82
C–D1.93.000.370.30
A–C7.63.203.020.44
Reviewer_48yzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether image quality can be read out from a single linear direction in frozen visual embeddings. It uses ridge regression to estimate ranking directions and examines quality ordering and spacing, feature geometry, and cross-dataset transfer. Experiments on nine IQA models, eight datasets, and several layers show that accurate linear ranking can coexist with weak quality alignment in feature distances. Covariance-weighted agreement helps explain the transfer results.

Critical strengths
  1. The paper makes an interesting distinction between linear quality prediction and the geometry of the original features. It shows that a model can rank image quality well even when its feature distances do not closely reflect quality differences.
  2. The experiments cover several model families, network depths, and distortion types, and include cross-dataset evaluation. The recurring observations across these settings support the main findings.
Critical weaknesses

The paper does not report a comparison with nonlinear regression on the same frozen features. Adding this baseline would help assess whether a single linear direction is sufficient.

Questions for authors
  1. Could the authors compare linear and nonlinear regression using the same frozen features?
  2. Do the main results remain stable across different training/test splits, particularly on TID2013?
2 comments
AuthorsOpenReview ↗

W1/Q1 - Linear vs nonlinear regression on the same frozen features

We thank the reviewer for the positive assessment and, we would like to to show the required analysis below.

W1/Q1 — Linear vs nonlinear regression on the same frozen features. Following Sonthalia et al. [1], we use a two-layer MLP (128 hidden units, ReLU, 30 hyper-parameter settings selected on validation SRCC) as an upper bound on the ordinal information in a representation, and train it on exactly the same frozen features (27 representations × 8 datasets).

SRCCLIVETID2013KADIDPIPALLIVECKonIQSPAQAGIQAW. mean
Linear (rank-axis)0.9510.7110.8580.4660.8200.8960.9140.8360.826
Nonlinear (MLP)0.9550.7800.8800.5170.8100.8950.9150.8300.841
Δ (Nonlinear − Linear)+0.004+0.069+0.022+0.051−0.010−0.001+0.001−0.006+0.015
Linear / Nonlinear0.9960.9130.9740.9011.0121.0010.9991.0070.978
Linear ≥ Nonlinear12/273/278/274/2722/2720/2712/2721/27102/216

W. mean: weighted by number of test images.

Analysis. The single linear direction captures 97.8% of the nonlinear upper bound and stays within 0.015 SRCC on average (weighted by test-set size); on the authentic and AI-generated datasets it is equal or better. A gap appears only on TID2013 and PIPAL, and it shrinks with layer depth (Δ = +0.019 at ℓ_early, +0.017 at ℓ_mid, +0.008 at ℓ_last). Nearly all the ordinal quality information is therefore linearly extractable. This table is added to the revised paper as Appendix A.3 (Table 5), with a paragraph in §5.1.

AuthorsOpenReview ↗

Q2 - Stability across train/test splits, in particular TID2013

The main tables keep the canonical split (either the official split where one exists or the PyIQA split otherwise). As a robustness check, we repeated the whole analysis on ten train/test splits per dataset, refitting every rank-axis on the unchanged features. For the datasets built from reference images (marked *), every split is reference-disjoint: all distorted versions of a reference image fall on the same side, as in the canonical split. For TID2013 (only 25 references) and AGIQA-3K we also ran a 5-fold cross-validation, disjoint by reference image and by text prompt, respectively.

Rankability per layer (R_ρ, mean of the 9 IQA encoders; paper = canonical split, Tables 3–4 and Fig. 2; splits = mean ± std over the 10 splits):

Datasetℓ_early paperℓ_early splitsℓ_mid paperℓ_mid splitsℓ_last paperℓ_last splits
LIVE*0.9470.940 ± 0.0070.9490.938 ± 0.0050.9560.948 ± 0.006
TID2013*0.6610.688 ± 0.0830.7020.733 ± 0.0700.7690.777 ± 0.065
— 5-fold, reference-disjoint0.681 ± 0.0690.711 ± 0.0720.771 ± 0.072
KADID-10k*0.8370.834 ± 0.0110.8520.849 ± 0.0100.8850.889 ± 0.009
LIVEC0.7820.820 ± 0.0200.8270.860 ± 0.0170.8520.871 ± 0.011
KonIQ-10k0.8850.888 ± 0.0030.8980.902 ± 0.0030.9040.908 ± 0.003
SPAQ0.9120.912 ± 0.0030.9150.916 ± 0.0030.9160.917 ± 0.003
AGIQA-3K0.8270.837 ± 0.0070.8390.852 ± 0.0080.8410.856 ± 0.008
— 5-fold, prompt-disjoint0.828 ± 0.0090.845 ± 0.0050.850 ± 0.003

Spacing, geometry and transfer at ℓ_last (paper → mean ± std over the 10 splits; paper values as in Tables 22–23):

DatasetRAISAnchor-QEAOrdering rateGlobal-QEASpearman(RAIS, R_ρ)Spearman(c_Σ, transfer)
LIVE*0.868 → 0.846 ± 0.0150.679 → 0.656 ± 0.0200.756 → 0.746 ± 0.0090.673 → 0.653 ± 0.0200.995 → 0.986 ± 0.0070.947 → 0.944 ± 0.012
TID2013*0.524 → 0.552 ± 0.0800.346 → 0.375 ± 0.0390.622 → 0.635 ± 0.0150.392 → 0.415 ± 0.0440.985 → 0.972 ± 0.0210.894 → 0.868 ± 0.054
— 5-fold, reference-disjoint0.543 ± 0.0930.375 ± 0.0340.635 ± 0.0130.417 ± 0.0330.963 ± 0.0280.843 ± 0.062
KADID-10k*0.700 → 0.712 ± 0.0190.472 → 0.484 ± 0.0080.677 → 0.683 ± 0.0030.465 → 0.481 ± 0.0090.997 → 0.993 ± 0.0050.933 → 0.941 ± 0.009
LIVEC0.708 → 0.712 ± 0.0120.432 → 0.439 ± 0.0120.657 → 0.659 ± 0.0050.464 → 0.461 ± 0.0170.985 → 0.985 ± 0.0050.965 → 0.966 ± 0.003
KonIQ-10k0.767 → 0.774 ± 0.0070.454 → 0.459 ± 0.0080.664 → 0.667 ± 0.0030.489 → 0.492 ± 0.0080.975 → 0.987 ± 0.0090.977 → 0.976 ± 0.001
SPAQ0.775 → 0.776 ± 0.0040.589 → 0.594 ± 0.0060.716 → 0.718 ± 0.0030.593 → 0.597 ± 0.0060.971 → 0.976 ± 0.0110.971 → 0.972 ± 0.001
AGIQA-3K0.708 → 0.734 ± 0.0130.411 → 0.433 ± 0.0140.644 → 0.653 ± 0.0050.457 → 0.479 ± 0.0130.969 → 0.977 ± 0.0110.964 → 0.968 ± 0.003
— 5-fold, prompt-disjoint0.729 ± 0.0130.432 ± 0.0090.653 ± 0.0030.479 ± 0.0120.970 ± 0.0200.967 ± 0.003

* Reference-disjoint splits (all distorted versions of a reference image on the same side).

Analysis.

  • Rankability is stable at every layer. The mean over splits stays within 0.04 of the paper at every layer (largest: LIVEC ℓ_early, +0.038) and within 0.02 at ℓ_last, with std ≤ 0.02 except TID2013. The depth trend is also preserved: ℓ_last is the most rankable layer on every dataset, in the paper and over the splits.
  • TID2013. Its standard deviation is larger than on the other datasets (± 0.065 at ℓ_last), yet the mean stays within 0.01 of the paper (0.777 vs 0.769), and so does the reference-disjoint 5-fold CV (0.771).
  • Spacing and geometry. RAIS and the three QEA measures change by at most 0.03. Spacing follows order (Spearman(RAIS, R_ρ) ≥ 0.96 on average over splits), and the native latent space stays weakly ordered (Anchor-QEA 0.35–0.66, same ranking of datasets).

Both tables are added to the revised paper as a new Appendix A.5 (Tables 8–9).

References [1] A. Sonthalia, A. Uselis, S. J. Oh. On the Rankability of Visual Embeddings. NeurIPS 2025.

Comments on the paper

2 comments

We sincerely thank all four reviewers for their time and constructive feedback. We are encouraged that the reviewers recognized several aspects of our work:

  • A new perspective on IQA representations. Reviewer BPXU noted that the paper offers a novel perspective on IQA through rankability and embedding geometry. Reviewer 48yz appreciated the distinction between linear quality prediction and the geometry of the original feature space.
  • New diagnostic tools. Reviewer BPXU recognized RAIS, Anchor-QEA, ordering rate, and Global-QEA as novel diagnostic tools. Reviewer DKch noted that the rank-axis captures both image ordering and the magnitude of quality differences.
  • Empirical findings. Reviewer UutJ found the findings that spacing follows order and rank-axis transfer are interesting.
  • Broad experimental evaluation. Reviewers 48yz, DKch, and BPXU appreciated the evaluation across nine IQA models, three network depths, eight benchmarks, multiple criteria, and cross-dataset transfer. Reviewer 48yz noted the consistency of the findings across these settings.

Dear Reviewers and Area Chairs,

We would like to clarify the positioning of our submission, which is in the Interpretability and Explainable AI track. Our objective is not to propose a new quality score predictor, but to investigate what perceptual quality information is encoded in the hidden representations of existing IQA models and how their latent space is structurally organized.

  1. Positioning. IQA models are compared almost exclusively by the SRCC/PLCC of their outputs with MOS, which cannot show to what extent representations are ordered by image quality scores. Our submission provides representation-level diagnostics for this purpose.
  2. Extensive findings. Across 9 IQA models, 3 layer depths and 8 benchmarks:
    1. Order: a single rank-axis is capable to order images by quality scores and beats the fine-tuned head in 52 of 70 encoder-dataset pairs.
    2. Geometry: the native embedding space is only weakly ordered based on image quality scores (Anchor-QEA 0.35).
    3. IQA vs generic encoders: Anchor-QEA reveals what IQA training adds to the latent space. At ℓlast\ell_{\text{last}}, the IQA encoders reach a mean Anchor-QEA of 0.45, against 0.09 for CLIP and 0.05 for DINOv2, a structural difference that the SRCC/PLCC of the outputs cannot show.
    4. Transfer: rank-axes transfer across datasets, which is explained by covariance-weighted agreement.
  3. Rebuttal additions. A nonlinear upper-bound (the rank-axis retains 97.8% of the MLP's performance), generic encoders (IQA training builds quality-score based ordering with layer-depth), ten reference-disjoint splits (stable results), inference cost (~10% of the FLOPs at an early layer) and a full-reference metrics comparison (the rank-axis outperforms LPIPS, DISTS and DeepDC), together with the suggested updates in the main paper and appendix.

We hope that the intended track and contribution will be considered in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.