acceptodds
Under review as a conference paper at ICLR 2027

Predicting the Frozen Ceiling: A Train-Free Diagnostic for Self-Supervised Retrieval

Abstract

How far can frozen self-supervised features go for strict nearest-neighbor retrieval, and can that limit be known without training? We show that the retrieval ceiling collapses to a single sufficient statistic—the signal-to-spread ratio of the per-query log-distance ratio, \(J=E[\log(b/a)]/Var(\log(b/a))\)—so that a closed-form diagnostic \(R@1=\Phi(J)\), computed from just five summary statistics with no training and no labels, recovers observed strict \(R@1\) to within 2.4 points (raw) and 5.2 points (readout) and ranks backbones at zero cost (ImageNet-100 Spearman 0.943). The diagnostic reveals that frozen DINOv2 features are linearly closed: no linear post-processor improves retrieval, and only a supervised nonlinear readout opens the space (0.688\(\to\)0.739). The ceiling is governed by the spread of the log-ratio, not by margins or dependence alone; raising dependence \(\rho\) is provably insufficient as shared query difficulty can drive \(\rho\to1\) without changing retrieval. The trained readout acts through regime-dependent mechanisms—\(\rho\)-driven on COCO, margin-driven on Stanford Dogs—and the theory-derived loss \(L_J\) underperforms SupCon when trained, exposing a prediction-optimization gap we interpret as a mechanistic finding. The result is a train-free, mechanism-aware diagnostic that reveals when frozen features are closed and why supervised readouts open them.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, Frozen representations, Nearest-neighbor retrieval, Margin-based analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4sCmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to estimate the performance ceiling of frozen self-supervised features in nearest-neighbor retrieval tasks without any training. The authors find that the retrieval performance of models such as DINOv2 is difficult to improve through conventional linear post-processing methods. By computing only five summary statistics, without requiring additional label information beyond a single labeled query set, the proposed method can predict the actual strict Recall@1, with prediction errors within 2.4 and 5.3 percentage points for raw and readout features, respectively. Furthermore, the method achieves a Spearman rank correlation of 0.943 on ImageNet-100.

Critical strengths

1.Firstly, I think this paper investigate an interesting practical question about how to characterize the retrieval capability of frozen self-supervised representations. And the proposed statistical diagnostic offers an interpretable perspective on nearest-neighbor retrieval performance and provide useful insights in this field. 2.Secondly, this paper shows solid experiments across plenty of datasets and different pretrained backbones like DINOv2, DINOv3, CLIP, and so on.The comparison between linear processing, unsupervised nonlinear transformations, and supervised readouts is also well presented. And the authors also show and analyse the improvement mechanisms on COCO and Stanford Dogs datasets which provides the insights as well.

Critical weaknesses

1.Firstly, the additional value of the proposed diagnostic is not sufficiently clear. The proposed diagnostic requires computing the nearest same-class and different-class distances for each query using labeled data. However, once these distances are available, the actual R@1 can be directly computed by comparing the two distances. Therefore, it is unclear what additional practical value the proposed approximation provides over direct retrieval evaluation, especially given its non-negligible prediction errors. 2.Secondly,the proposed closed-form predictor relies on distributional assumptions that are not fully supported by experiment. In particular, the paper reports that the fitted marginal distributions and Gaussian copula are rejected by goodness-of-fit tests, while the prediction errors become substantially larger on Stanford Dogs, reaching approximately 7–19 percentage points.

Questions for authors
  1. Could the authors clarify the specific use cases where the proposed diagnostic provides advantages over directly computing R@1 from the same nearest-neighbor distances? In particular, can it provide useful information for model selection that cannot be obtained from direct retrieval evaluation? Just give some specific occasions. 2.The paper shows that the fitted marginal distributions and Gaussian copula are rejected by goodness-of-fit tests, could the authors clarify why the closed-form predictor remains reliable, and under what conditions its prediction accuracy can be expected to hold across different datasets and models?
Reviewer_doZ6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study the problems of (1) evaluating the retrieval performance of frozen features and (2) estimating that performance without training. They first show that the retrieval performance of frozen DINO features can hardly be improved by linear post-processors, suggesting that this performance is a limit of frozen DINO features. They then propose a statistic to predict this limit. Finally, they conduct experiments to verify their claims.

Critical strengths
  • The problem is interesting and well-motivated.
  • The proposed method is simple and effective.
  • The experiments are extensive.
Critical weaknesses
  • The paper is not well-written:
    • The Introduction does not clearly state the problems of interest, and does not discuss how related works address these works.
    • Some notations are used before they are defined, e.g., ρ\rho and ww.
    • The experiment settings are unclear, e.g., the methods used in Table 1 are not properly introduced.
  • The proposed approach is not well-justified. In order to compute the proposed statistic, we need to compute aa and bb for each query, which is sufficient for computing strict R@1
  • Some claims are misleading. For example, the paper claims that the proposed statistic does not require training. But in Table 3, in order to predict the performance of the readout, we still need to train the head first.
Questions for authors
  • Questions (The questions are mainly based on the weaknesses)
    • How do related works address the problems of interest?
    • What are the baselines in Table 1?
    • Why don't we directly calculate R@1, but estimate it with the proposed statistic?
    • How are the statistics in Table 3 calculated?
  • Other issues
    • The paper does not follow the formatting instructions. For example, the table captions are below the tables.
Reviewer_MWUAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper deals with the assessment of retrieval performance in a pre-trained space. The theoretical part is not needed for the experiments and has no impact on the conclusions. The paper presents simple experiments on the impact linear transforms on DINOv2 and CLIP spaces with artificial, needless complexity

Critical strengths

The paper shows that for some datasets and some pretrained spaces, applying a set of arbitrarily chosen linear transforms (see table 1) does not improve recall at 1.

Critical weaknesses

The paper introduces assumptions about various distributions, mainly the per-query log-distance ratio, to propose a diagnostic estimate of recall at one. The submission fails to explain, why all of this is needed, if recall at one can be trivially, efficiently estimated directly, with no assumptions.

The "basic unanswered questions" is claimed to be: how far can a frozen embedding actually go for strict retrieval, and can that limit be known before any training is spent? This is clearly data dependent (for exactly the same objects, there could be many different retrieval problems), and the answer can be thus obtained for the particular setup, which is of limited interest.

All the results presented in the paper hold exactly for the dataset, space and set-up chosen. There is no generalization.

Questions for authors

Explain, why a user should not directly estimate R@1 and instead use the probabilistic models used

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.