- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to estimate the performance ceiling of frozen self-supervised features in nearest-neighbor retrieval tasks without any training. The authors find that the retrieval performance of models such as DINOv2 is difficult to improve through conventional linear post-processing methods. By computing only five summary statistics, without requiring additional label information beyond a single labeled query set, the proposed method can predict the actual strict Recall@1, with prediction errors within 2.4 and 5.3 percentage points for raw and readout features, respectively. Furthermore, the method achieves a Spearman rank correlation of 0.943 on ImageNet-100.
Critical strengths
1.Firstly, I think this paper investigate an interesting practical question about how to characterize the retrieval capability of frozen self-supervised representations. And the proposed statistical diagnostic offers an interpretable perspective on nearest-neighbor retrieval performance and provide useful insights in this field. 2.Secondly, this paper shows solid experiments across plenty of datasets and different pretrained backbones like DINOv2, DINOv3, CLIP, and so on.The comparison between linear processing, unsupervised nonlinear transformations, and supervised readouts is also well presented. And the authors also show and analyse the improvement mechanisms on COCO and Stanford Dogs datasets which provides the insights as well.
Critical weaknesses
1.Firstly, the additional value of the proposed diagnostic is not sufficiently clear. The proposed diagnostic requires computing the nearest same-class and different-class distances for each query using labeled data. However, once these distances are available, the actual R@1 can be directly computed by comparing the two distances. Therefore, it is unclear what additional practical value the proposed approximation provides over direct retrieval evaluation, especially given its non-negligible prediction errors. 2.Secondly,the proposed closed-form predictor relies on distributional assumptions that are not fully supported by experiment. In particular, the paper reports that the fitted marginal distributions and Gaussian copula are rejected by goodness-of-fit tests, while the prediction errors become substantially larger on Stanford Dogs, reaching approximately 7–19 percentage points.
Questions for authors
- Could the authors clarify the specific use cases where the proposed diagnostic provides advantages over directly computing R@1 from the same nearest-neighbor distances? In particular, can it provide useful information for model selection that cannot be obtained from direct retrieval evaluation? Just give some specific occasions. 2.The paper shows that the fitted marginal distributions and Gaussian copula are rejected by goodness-of-fit tests, could the authors clarify why the closed-form predictor remains reliable, and under what conditions its prediction accuracy can be expected to hold across different datasets and models?