acceptodds
Under review as a conference paper at ICLR 2027

Rank Is Not Quality: Label-Free Model Selection Inverts under Instance-Private Nuisance

Abstract

Self-supervised encoders are increasingly selected without labels, most often by the effective rank of their embeddings (RankMe), on the premise that a more spread-out embedding is a more useful one; anti-collapse objectives such as VICReg and LeJEPA push the same quantity up during training. Because such criteria are used where labels are absent, a failure goes unnoticed unless labels are brought back. We show that effective rank can rank models backwards. When a label-irrelevant, instance-specific panel is appended beside each CIFAR-10 image, effective rank rises (9/9 paired runs) while linear-probe accuracy falls from to . Across that population RankMe's correlation with accuracy is (), it inverts within each of SimCLR, VICReg and BYOL, and a preregistered 30-run replication reproduces the inversion before convergence ( at epoch 5). A first proposition shows why the objectives cannot rule this out: an encoder that whitens (and, where required, Gaussianises) instance-private content attains the alignment optimum and the optimum of the marginal health terms in use, so no such objective can prefer semantic content. Nor is the failure specific to rank. On a preregistered dial from shared to instance-private nuisance, every criterion we measured prefers a nuisance-bearing model to the clean one somewhere: spread criteria from mid entropy upward, cluster criteria such as CLID's cluster learnability wherever the nuisance is shared; a second proposition shows that no criterion reading only the embedding distribution can be guaranteed safe. Three of the dial's five preregistered tests failed, as did an anti-collapse term designed to fix the problem; we report them all, claim no graded inversion, and restricted our claim to instance-private nuisance only after seeing them. The accuracy collapse itself is Chen et al.'s RandBit result; its consequence for label-free selection is, to our knowledge, new. Where candidate models may differ in what they capture, rank should not select among them, and no distribution-only criterion replaces a labelled check.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, label-free model selection, effective rank, joint-embedding architectures, representation evaluation, shortcut learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yWqHOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper demonstrates that a high effective rank in self-supervised learning does not necessarily indicate a good representation. When label-independent, instance-specific nuisance is added to CIFAR-10, the model may learn this nuisance instead of meaningful semantic information; while the rank increases, downstream accuracy can drop significantly. The authors argue that since label-free metrics that only examine the distribution or structure of embeddings may fail to distinguish between semantic and nuisance information, it is risky to judge model quality based solely on rank.

Critical strengths
  1. Presents a clear and compelling counterexample where effective rank increases while downstream accuracy collapses across SimCLR, VICReg, and BYOL
  2. Extends the empirical finding with theoretical propositions showing the fundamental limitations of distribution-only label-free metrics
  3. Demonstrates strong research transparency by reporting failed preregistered tests, incorrect predictions, and post-hoc narrowing of the claims
Critical weaknesses
  1. The scope of the experiment is somewhat limited. Since the key results are based on CIFAR-10, ResNet-18, and an artificial “nuisance panel” unaffected by augmentation, it remains to be confirmed whether the same phenomenon occurs in naturally occurring shortcuts, large-scale datasets such as ImageNet, and ViT-family architectures.
  2. The strongest form of the generalization claim was not sufficiently supported by the preregistration experiments. In the entropy dial experiments, 3 out of 5 preregistered tests failed, and in particular, the key hypothesis—that a sufficiently strong negative correlation would emerge between RankMe and accuracy—was not satisfied.
  3. While the theoretical results reveal important limitations, they do not fully explain the actual learning dynamics. Proposition 1 shows that representations using only nuisance features can be optimal solutions to the objective under whitening or Gaussianization conditions, but it does not explain why or how often SGD actually converges to such solutions.
Questions for authors

The Introduction is too brief and does not adequately explain the research field or provide sufficient background on the motivation. The “Contribution” section needs to be summarized using fewer bullet points, and Sections 2, 3, and 4 are unnecessarily fragmented, which reduces readability. Is there a specific reason why it was written this way?

1 comment
AuthorsOpenReview ↗

We agree on scope, the failed tests and the theory's limit; why the paper is structured as it is

Thank you. We agree with all three weaknesses.

W1. Scope. Agreed; this is the first limitation in Section 10. We have no result on ImageNet, ViTs or a naturally occurring shortcut, and we do not claim one.

On augmentation we can add one result. It comes from an earlier preregistered study on the same construction; Appendix F mentions the study, but its numbers are not in the paper. That study gave each view its own fresh panel, which is what augmentation does when it destroys the nuisance. The panel is then not captured: accuracy and effective rank stay at the clean level. SimCLR reaches 0.780 with rank 221, against 0.780 and 228 clean (3 seeds). With the panel shared by both views, SimCLR's accuracy is 0.102 and its rank is 440. So the inversion we report needs nuisance that survives augmentation. Augmentation that reaches the panel only partly is untested.

W2. The preregistered tests. Agreed, and the paper draws the same conclusion. Three of the dial's five tests failed (Section 6, Table 4), including the headline test (ρ=−0.208\rho = -0.208, p=0.11p = 0.11, against a preregistered −0.4-0.4). We therefore claim no graded inversion.

The claim we keep is the contrast between clean and private-nuisance models:

  • effective rank is 1.49×1.49\times higher in 9/9 paired runs while accuracy falls from 0.782 to 0.115;
  • RankMe's correlation with accuracy is ρ=−0.686\rho = -0.686 over those runs, and −0.684-0.684 at epoch 5 in the preregistered replication with 30 runs.

The paper says that this narrowing was made after seeing the dial. What the dial does show changes the practical advice. Cluster criteria fail at the shared end (CLID's cluster learnability is 1.000 where accuracy is 0.161), so switching from rank to a cluster criterion does not escape the problem.

W3. Theory and learning dynamics. Agreed. Proposition 1 is an existence statement: a whitened instance-private code attains the optimum of these objectives, so they cannot prefer semantic content. It does not say how often SGD reaches such a solution. Our preregistered test that trained encoders move toward SIGReg's optimum failed (Appendix B).

The only objective whose training logs show a path is BYOL, which lies outside Proposition 1. With private panels, its accuracy falls between epochs 10 and 30 (0.405 → 0.348 in one seed, 0.339 → 0.248 in the other), while its effective rank stays flat (301 → 314; 341 → 339). That is consistent with capture growing during training, but it is an observation, not an explanation, and it does not test the proposition.

Q. Why the structure. The paper grew from a sequence of preregistered studies. We kept each one in its own section so that every verdict stays next to its preregistration. You are right that this fragments the argument and leaves the Introduction thin. A revision would merge Sections 2–4 into one section on the nuisance axis and the inversion, cut the contributions from six to three, and open with where label-free selection is used and why its failures go unnoticed without labels.

Reviewer_ezBPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of selecting a self-supervised model without using labels. The authors claim that the existing methods, such as RankMe, are easily influenced or even misled by nuisance information. To study this, the authors add a panel to the right of the image to add the nuisance information. The results show that due to the influence of the nuisance information, existing label-free selection methods select the models that have low linear-probe accuracy.

Critical strengths
  1. The research question studied in this paper is interesting and important. The criteria in label-free model selection methods are not reliable and can be easily influenced by nuisance.
  2. The authors conduct experiments to verify the claim. With the panel, the effective rank increases while the linear-probe accuracy decreases dramatically.
  3. The authors conduct experiments on various methods (SimCLR, VICReg, BYOL) and criteria (RankMe, \alpha-ReQ, CLID’s cluster learnability).
Critical weaknesses
  1. The paper's organization can be improved. The authors may consider reorganizing the structure of the paper to improve the presentation.
  2. The phenomenon that nuisance leads the SSL model to learn irrelevant features and leads to performance degradation has been studied before [1].
  3. The authors introduce a different setting to verify their claims. In the setting of this paper, the authors do not apply data augmentation to the panel. In standard SSL training, data augmentation is applied to the whole images. Thus, the nuisance information will not remain unchanged or even be removed.
  4. The authors only conduct experiments on a small image dataset, e.g., cifar-10. Experiments on larger datasets, e.g., ImageNet, are required to verify the generalization of the method.

[1] Intriguing Properties of Contrastive Losses

Questions for authors

If data augmentation is applied to the whole input, including the panel, will the ranking given by Rankme still be inverted?

Can the same phenomenon be verified on larger datasets?

1 comment
AuthorsOpenReview ↗

When augmentation destroys the panel, it is not captured; scale is untested; what is new over Chen et al.

Thank you for the two direct questions.

Q1 / W3. Augmentation applied to the panel. You are right that the panel is outside the augmentation. It is appended after the image is augmented (Section 2), so both views carry the same panel. That is the condition our claim needs.

We can answer the extreme case of your question from an earlier preregistered study on the same construction. Appendix F mentions it, but its numbers are not in the paper. That study gave each view its own fresh panel, which is what augmentation does when it fully destroys the nuisance: the panel's pixel statistics are unchanged, but the two views no longer share it. With 8 squares, the panel is not captured. Accuracy and effective rank stay at the clean level (3 seeds each, epoch 30):

  • SimCLR: accuracy 0.780, rank 221 (clean: 0.780, 228)
  • VICReg: 0.766, 206 (clean: 0.769, 214)
  • BYOL: 0.629, 230 (clean: 0.637, 228)

The same panel shared by both views gives SimCLR an accuracy of 0.102 and a rank of 440. So the rank rise comes from the encoder capturing the panel, not from the extra input, and a panel that is not captured adds no inversion of its own.

To answer your question about the ranking exactly: pooled across the three objectives, RankMe's correlation with accuracy in this study is still negative (ρ=−0.41\rho = -0.41, n=36n = 36 clean and fresh-panel runs). That comes from differences between objectives, as on our clean populations (Section 4), not from the panel; within each objective the sign is not consistent.

We have not tested the middle case: augmentation that reaches the panel only partly (a crop that keeps part of it, a flip that moves it). Our claim is therefore limited to nuisance that survives augmentation. That is also the condition of Chen et al.'s RandBit construction ("not altered by augmentation").

Q2 / W4. Larger datasets. We agree; it is the first limitation in Section 10. We have no result beyond CIFAR-10 and ResNet-18. Chen et al. report that the accuracy collapse itself persists on ImageNet. Whether the selection inversion does is untested, and we do not claim it.

W2. Relation to Chen et al. (2021). The suppression phenomenon is theirs, and the paper says so in the abstract and in Section 1 ("What is and is not ours": "The accuracy collapse under instance-private content is not our finding"). What we add is its consequence for choosing models without labels, which they did not measure:

  1. RankMe ranks the models backwards: ρ=−0.686\rho = -0.686 (p=0.002p = 0.002) over 18 runs, and ρ=−0.684\rho = -0.684 at epoch 5 in a preregistered replication with 30 runs.
  2. Proposition 1 shows that objectives with an explicit marginal health term cannot prefer semantic content, because a whitened instance-private code is also optimal. It is an existence result, not a claim about training.
  3. On the entropy dial (Section 6), cluster criteria fail at the shared end instead: CLID's cluster learnability reaches 1.000 where accuracy is 0.161. Proposition 2 shows that no criterion reading only the embedding distribution can be guaranteed safe.

W1. Organisation. We agree. The paper grew from a sequence of preregistered studies, and we kept each one next to its verdict, which fragments the argument. A revision would merge Sections 2–4, cut the contributions to three and give the Introduction a proper motivation.

Reviewer_TPXdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of label-free model selection criteria for self-supervised representations, particularly whether effective rank can reliably reflect downstream representation quality. The authors construct an instance-private nuisance setting by appending label-irrelevant panels to CIFAR-10 images and evaluate three self-supervised learning methods: SimCLR, VICReg, and BYOL. Experimental results show that effective rank increases by 1.49× while linear-probe accuracy decreases from 78.2% to 11.5%, producing a negative correlation between RankMe and downstream accuracy. Overall, the work highlights an important limitation of label-free representation evaluation, although the practical generality of its conclusions remains insufficiently established.

Critical strengths
  1. The paper raises an important concern about label-free representation evaluation: a higher effective rank may not translate into better downstream performance. Its focus on model-selection rankings, rather than representation collapse alone, helps assess how reliably unsupervised metrics can guide model selection.

  2. The experimental design provides clear evidence of ranking inversion under controlled nuisance conditions, where the authors evaluate SimCLR, VICReg, and BYOL using paired experiments and demonstrate that effective rank increases even when downstream classification accuracy approaches chance level. The additional 30-run replication further supports the consistency of this phenomenon across different training stages.

  3. The two propositions provide formal explanations for the limitations of marginal distribution-based objectives and evaluation criteria under their stated assumptions. Furthermore, the authors explicitly report unsuccessful pre-registered experiments and distinguish confirmed findings from exploratory observations, improving the transparency and credibility of the analysis.

Critical weaknesses
  1. As acknowledged in Section 10, RankMe was primarily proposed for comparing different training configurations of the same method on a fixed dataset, whereas the central experiments compare models trained on clean images against models trained on modified images containing instance-private nuisance. Consequently, the observed inversion may not directly reflect the behavior of RankMe in its standard model-selection setting. Although the authors provide within-method comparisons, these still involve changes to the input distribution. It would be valuable to conduct controlled experiments in which all candidate models are trained on the same nuisance-containing dataset but differ in hyperparameters, training checkpoints, or regularization strengths. This would clarify whether RankMe also produces incorrect selections under practically relevant fixed-data conditions.

  2. Section 6 reports that three of the five pre-registered tests failed, including the anticipated negative correlation between RankMe and accuracy across the entropy dial (ρ=−0.208ρ = −0.208, p=0.11p = 0.11). Furthermore, the accuracy degradation is not consistently graded, and the transition to b=16b = 16 changes both nuisance entropy and whether the test panels were observed during training. These factors make it difficult to isolate the effect of nuisance entropy on metric behavior. Although the authors appropriately narrow their main claim, stronger conclusions about the boundaries of different evaluation criteria would require a more controlled design. I suggest independently varying nuisance entropy and train–test panel novelty, increasing the number of seeds, and reporting within-method trends and uncertainty intervals to determine which effects remain statistically reliable.

  3. The experiments use a LiDAR-style implementation and only the cluster-learnability component of CLID, rather than consistently evaluating the complete original metrics. Since the paper aims to characterize the limitations of label-free model selection, these omissions make it difficult to determine whether the observed failures are specific to the selected implementations or representative of existing evaluation approaches more broadly. I recommend including the original implementations of LiDAR and full CLID, together with additional recent criteria such as IdEst, under consistent feature-sampling and evaluation protocols. This would provide a more fair assessment.

  4. Proposition 1 establishes that a whitened instance-private representation can satisfy certain alignment and marginal-distribution objectives under idealized assumptions, including sufficient nuisance dimensionality and the ability of the encoder to whiten or Gaussianize the nuisance. However, this existence result does not establish that practical optimization procedures will converge toward such representations. The paper also acknowledges that BYOL lies outside the proposition and that its pre-registered SIGReg experiment did not confirm the anticipated behavior of trained encoders. I suggest complementing the theoretical analysis with experiments that explicitly track nuisance information, semantic information, and marginal regularization terms throughout training. Direct evaluations of additional relevant objectives, particularly Barlow Twins or LeJEPA, would also strengthen the connection between the theoretical claims and practical self-supervised learning.

Questions for authors

Please carefully checking weaknesses.

1 comment
AuthorsOpenReview ↗

Fixed data: RankMe fails across objectives, not within one; what the dial and the theory do not show

Thank you for the close reading. We agree with the substance of all four points. Numbers below that are not printed in the paper are recomputed from the per-run files in the supplement, with no new training.

W1. Fixed-data selection. The dial already contains fixed-data populations. At b=16b = 16, every model is trained on the same private-panel dataset. Choosing among its 18 candidates (3 objectives × 2 seeds × 3 checkpoints), RankMe picks a model with accuracy 0.098 when 0.405 is available (ρ=−0.313\rho = -0.313). At b=2b = 2 it picks 0.570 against 0.759.

These are comparisons across objectives: the 0.098 pick is a SimCLR model, and the 0.405 is BYOL. Within a single objective, which is the setting RankMe was proposed for, it is nearly right. Its pick is the most accurate model in 8 of 18 (objective, bb) cells, the median shortfall in accuracy is 0.0008, and the worst is 0.099.

So the failure needs candidates that differ in what they capture. Seeds and checkpoints of one objective usually differ little in what they capture. The exception is BYOL at b=16b = 16, where accuracy falls during training while rank stays flat (W4 below). There RankMe's pick is 0.065 below the best (0.339 against 0.405). That is the nearest we have to your test, which needs one objective on one dataset with a hyperparameter that controls capture. We have not run it, so the claim stays as Section 10 states it.

W2. The dial. We agree that it does not isolate entropy, and the paper claims no graded inversion.

  • From b=6b = 6 to b=8b = 8, test panels come from the training alphabet at both levels. Even so, rank and accuracy already move in opposite directions there: rank +102, accuracy down, in 6/6 units. At the step where novelty changes (b=8→16b = 8 \to 16), accuracy does not move (+0.007; Section 6). But over all shared levels the correlation is +0.17, so this is one step, not a trend.
  • Within each objective on the preregistered run (n=12n = 12 each, 95% bootstrap intervals over runs), RankMe's correlation with accuracy is SimCLR −0.21-0.21 [−0.78,+0.60][-0.78, +0.60], VICReg −0.49-0.49 [−0.93,+0.36][-0.93, +0.36] and BYOL −0.10-0.10 [−0.63,+0.69][-0.63, +0.69]. All three intervals include zero.

This is why the claim we keep is the contrast between the two ends of the dial. More seeds and a control with seen versus novel test panels would be the right next step; we do not have them.

W3. Original LiDAR, full CLID, IdEst. Agreed. The paper labels its versions "LiDAR-style" and "cluster learnability (CL of CLID)" and says that IdEst was not measured (Section 9). We have no results with the original implementations, so the paper's statements apply only to the versions we measured.

W4. From existence to dynamics. We agree. Proposition 1 shows that these objectives cannot prefer semantic content. It does not show that SGD reaches the nuisance optimum. Our preregistered test that trained encoders move toward SIGReg's optimum failed: ρ=−0.427\rho = -0.427 against a preregistered −0.5-0.5, on pixel-JEPA embeddings (Appendix B).

The only objective whose logs show a path is BYOL, which lies outside Proposition 1. At b=16b = 16 its accuracy falls during training (0.405 → 0.369 → 0.348 and 0.339 → 0.275 → 0.248 at epochs 10, 20 and 30), while its rank stays flat (301 → 314; 341 → 339). This is consistent with capture growing during training, but it does not test the proposition. SimCLR and VICReg are already at chance by epoch 10. We did not log a measure of nuisance information or the separate loss terms during training, and we did not train Barlow Twins or LeJEPA objectives.

A correction of our own. SimCLR's within-objective correlation in Figure 1b (−0.771-0.771, p=0.10p = 0.10, n=6n = 6) is not significant. The abstract, Contribution 2 and Section 4 should therefore not say that RankMe inverts "within each" objective on correlation alone. The test that carries this claim is the paired one: rank is higher in the captured model in 9/9 (method, seed) units (exact sign-flip p=0.0039p = 0.0039).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.