- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper demonstrates that a high effective rank in self-supervised learning does not necessarily indicate a good representation. When label-independent, instance-specific nuisance is added to CIFAR-10, the model may learn this nuisance instead of meaningful semantic information; while the rank increases, downstream accuracy can drop significantly. The authors argue that since label-free metrics that only examine the distribution or structure of embeddings may fail to distinguish between semantic and nuisance information, it is risky to judge model quality based solely on rank.
Critical strengths
- Presents a clear and compelling counterexample where effective rank increases while downstream accuracy collapses across SimCLR, VICReg, and BYOL
- Extends the empirical finding with theoretical propositions showing the fundamental limitations of distribution-only label-free metrics
- Demonstrates strong research transparency by reporting failed preregistered tests, incorrect predictions, and post-hoc narrowing of the claims
Critical weaknesses
- The scope of the experiment is somewhat limited. Since the key results are based on CIFAR-10, ResNet-18, and an artificial “nuisance panel” unaffected by augmentation, it remains to be confirmed whether the same phenomenon occurs in naturally occurring shortcuts, large-scale datasets such as ImageNet, and ViT-family architectures.
- The strongest form of the generalization claim was not sufficiently supported by the preregistration experiments. In the entropy dial experiments, 3 out of 5 preregistered tests failed, and in particular, the key hypothesis—that a sufficiently strong negative correlation would emerge between RankMe and accuracy—was not satisfied.
- While the theoretical results reveal important limitations, they do not fully explain the actual learning dynamics. Proposition 1 shows that representations using only nuisance features can be optimal solutions to the objective under whitening or Gaussianization conditions, but it does not explain why or how often SGD actually converges to such solutions.
Questions for authors
The Introduction is too brief and does not adequately explain the research field or provide sufficient background on the motivation. The “Contribution” section needs to be summarized using fewer bullet points, and Sections 2, 3, and 4 are unnecessarily fragmented, which reduces readability. Is there a specific reason why it was written this way?
We agree on scope, the failed tests and the theory's limit; why the paper is structured as it is
Thank you. We agree with all three weaknesses.
W1. Scope. Agreed; this is the first limitation in Section 10. We have no result on ImageNet, ViTs or a naturally occurring shortcut, and we do not claim one.
On augmentation we can add one result. It comes from an earlier preregistered study on the same construction; Appendix F mentions the study, but its numbers are not in the paper. That study gave each view its own fresh panel, which is what augmentation does when it destroys the nuisance. The panel is then not captured: accuracy and effective rank stay at the clean level. SimCLR reaches 0.780 with rank 221, against 0.780 and 228 clean (3 seeds). With the panel shared by both views, SimCLR's accuracy is 0.102 and its rank is 440. So the inversion we report needs nuisance that survives augmentation. Augmentation that reaches the panel only partly is untested.
W2. The preregistered tests. Agreed, and the paper draws the same conclusion. Three of the dial's five tests failed (Section 6, Table 4), including the headline test (ρ=−0.208, p=0.11, against a preregistered −0.4). We therefore claim no graded inversion.
The claim we keep is the contrast between clean and private-nuisance models:
The paper says that this narrowing was made after seeing the dial. What the dial does show changes the practical advice. Cluster criteria fail at the shared end (CLID's cluster learnability is 1.000 where accuracy is 0.161), so switching from rank to a cluster criterion does not escape the problem.
W3. Theory and learning dynamics. Agreed. Proposition 1 is an existence statement: a whitened instance-private code attains the optimum of these objectives, so they cannot prefer semantic content. It does not say how often SGD reaches such a solution. Our preregistered test that trained encoders move toward SIGReg's optimum failed (Appendix B).
The only objective whose training logs show a path is BYOL, which lies outside Proposition 1. With private panels, its accuracy falls between epochs 10 and 30 (0.405 → 0.348 in one seed, 0.339 → 0.248 in the other), while its effective rank stays flat (301 → 314; 341 → 339). That is consistent with capture growing during training, but it is an observation, not an explanation, and it does not test the proposition.
Q. Why the structure. The paper grew from a sequence of preregistered studies. We kept each one in its own section so that every verdict stays next to its preregistration. You are right that this fragments the argument and leaves the Introduction thin. A revision would merge Sections 2–4 into one section on the nuisance axis and the inversion, cut the contributions from six to three, and open with where label-free selection is used and why its failures go unnoticed without labels.