acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Encoders Learn a Task Prior in Their Spectrum That Isotropy Erases

Abstract

Self-supervised objectives increasingly push representations toward an isotropic Gaussian, which LeJEPA proves optimal for worst-case linear probes. We show that once tasks come from a distribution, a representation's spectrum is a label-free prior over them, and isotropy is the uniform prior. A one-penalty ridge probe on spectrum is generalised ridge on the latents with per-direction penalty , and is the Bayes (Wiener) estimator iff , the prior's variances, for any sample size, dimension and design that excites every direction (Lindley–Smith read from the representation's side), and a power-law prior recovers the spectral-bias learning-curve rate. Theorem A prices a spectrum in labels, and Theorem B shows that a -class task's prior has rank , so isotropy flattens exactly the directions that carry the task. Under a protocol on 23 encoders and 15 vision and text tasks (169 cells), isotropy costs a median 21 accuracy points at 200 labels (range 0.5–64), and recovering the discarded prior takes a median 500 labels, a price predicted () by the cost of detecting the weakest task direction, found post hoc on 20 cells, then replicated on the other 149. The prior comes back without labels. Averaged over other tasks, it recovers a median 89% of the oracle gain, and training toward a non-flat spectral target instead of isotropy raises 200-label accuracy from 60% to 80%, with the isotropy-trained encoder still worse even given the oracle prior. A supervised control pays the same price, so the effect is a property of linear probing itself.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, isotropy, JEPA, spectral regularization, linear probing, ridge regression, Bayesian prior, spectral bias, label efficiency, few-shot transfer, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QzmwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the impact of isotropy latents on downstream tasks, showing that it comes at the expanse of degrading performance. They also show that matching the task's prior can benefit.

Critical strengths
  • Tackle the latent geometry is important task. Especially well studied world models.
Critical weaknesses
  • writing level way below the standard of a top-tier conference. I read the abstract several times and didnt catch almost anything. several examples: 1) using unexplained notations, which I think that should exists in the abstract at all unless they emphasize something - e.g., \lambda, what is it? s_i - what is the i stands for? indexing of what? 2) using vague terms - what cells stands for? what is it a rank of a task? 3) unclear parts: " Theorem A prices a spectrum in labels" - dont understand what is that means. "Under a protocol on 23 encoders and 15 vision and text tasks" - on which protocol are you talking here? 4) The most important things, I've read the abstract several times and cannot tell what is the general purpose of the paper. The low level writing is continue to the intro: "None puts a distribution over the downstream tasks. This paper does. Put a prior w ∼ (0, P) on the linear read-outs a representation will be asked for." - what is "put distribution" means?

  • You are using notations without first declare them - lines 45-46 - what are nn and dd? what does vv in line 48 means?

Questions for authors

In the abstract you wrote Theorom A (line 19) and in the intro you write Theorom 1 (lines 45-46). are they both the same? If they dont then why do you have two metrics for numbering? because two lines after you are referring to Theorom A.

  • line 84 - "isotropising" - I dont think it is a valid word in English.

  • Where is the Conclusions section? I think it is important to summarize such paper.

  • Think that with respect to whitening, this paper is relevant here [1].

I think that even if the paper is super smart and tackle useful task, myself - as a human - cannot catch the idea since it is too involved with inaccurate English, non coherent logical flow, heavy mathematical derivation without explanations. Definitions and notations should first introduced then used, you should add some intuition for derivations. thus, even after trying hard, reading the abstract and intro several times, trying to follow the Theory part I am still more confused than learned. So sorry but I think that the paper should be rewritten aggressively in a more understandable way, and then be submitted again.

refs:

[1] Whitened CLIP as a Likelihood Surrogate of Images and Captions. Betser et al. ICML 25

Reviewer_aSmGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether the anisotropic spectrum of self-supervised representations encodes a useful prior over downstream tasks, challenging the view that isotropic representations are universally preferable. The authors show that applying standard ridge regression to spectrally rescaled features is equivalent to using direction-dependent regularization, and that matching the representation spectrum to the covariance of downstream task weights minimizes Bayesian prediction risk under their assumptions. They further derive results relating task priors, spectral structure, and the number of labels required to recover a discarded prior.

The authors evaluate these predictions across 169 encoder–task combinations involving vision and text representations. They report that isotropizing pretrained features causes a median reduction of 21 percentage points in few-shot accuracy at 200 labels, while task-matched or transferred spectral priors can recover substantial performance. Finally, controlled training experiments with VICReg on CIFAR-10 show that non-isotropic spectral targets substantially outperform an isotropic target, suggesting that representation anisotropy can be beneficial for label-efficient downstream learning.

Critical strengths

S1. The paper provides a principled interpretation of representation anisotropy as implicit direction-dependent regularization in few-shot linear probing. Beyond the known connection to Bayesian ridge regression, Theorems A and B investigate the label cost of recovering task-relevant directions and the structure of multiclass task priors. These results offer a useful perspective on when isotropy may be suboptimal, particularly in label-limited settings.

S2. The authors evaluate their hypotheses across 169 encoder–task combinations, covering multiple SSL objectives, architectures, and vision and text datasets. The observed loss in few-shot accuracy after isotropization is substantial, and the proposed label-cost predictors correlate strongly with empirical measurements. Importantly, Section 5.5 moves beyond post-hoc feature rescaling and demonstrates that training toward non-isotropic spectra can improve few-shot performance, providing preliminary evidence of practical relevance.

Critical weaknesses

W1. The empirical results do not sufficiently establish that enforcing isotropy during SSL training destroys useful task information.

  • W1.1. Post-hoc rescaling versus representation learning: Most experiments (§4–§5) modify the spectra of frozen representations before ridge probing. These interventions change the effective direction-dependent regularization without changing the information encoded in the features. Consequently, the observed performance degradation primarily establishes the sensitivity of ridge probes to feature scaling, rather than demonstrating that isotropic SSL objectives inherently destroy task-relevant information.

  • W1.2. The central training intervention (§5.5) is restricted to VICReg ResNet-18 on CIFAR-10. Although the reported gains are substantial, they are insufficient to establish that non-isotropic spectral targets consistently improve SSL across architectures, objectives, and downstream tasks.

  • W1.3. The paper's broader motivation concerns the spectral structure that SSL encoders should learn, whereas most of the empirical evidence concerns the effect of feature rescaling on a particular downstream probe.

W2. The proposed task-prior interpretation and its practical advantage over simpler spectral regularization remain insufficiently established.

  • W2.1. The optimality result in Theorem 1 relies on a linear prediction model with a single ridge penalty and a task prior aligned with the representation coordinates. Although the mathematical result is meaningful under these assumptions, it remains unclear how strongly its prescription applies to other downstream adaptation mechanisms.

  • W2.2. Task-specific prior versus generic anisotropy: In §5.5, a simple label-free power-law spectrum si∝i−1s_i\propto i^{-1} achieves essentially the same performance as the estimated population task prior. This raises an important question about whether the gains arise from matching a meaningful distribution of task importance or simply avoiding an isotropic spectral target.

  • W2.3. Without a clearer distinction between task-prior alignment and generic spectral shaping, the evidence does not fully establish the practical necessity of the proposed task-prior framework.

Suggestion: The authors may consider discussing Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning (ICML 2026), which provides generalization bounds connecting few-shot transfer to within-class variance along task-discriminative directions.

Questions for authors

Q1: Can the authors demonstrate that non-isotropic spectral regularization improves few-shot transfer beyond VICReg on CIFAR-10, preferably using a different SSL objective or held-out downstream task?

Q2: Given that the label-free i−1i^{-1} spectrum matches the estimated population prior in Section 5.5, what evidence establishes that task-prior alignment is necessary rather than generic anisotropy?

Reviewer_EPV5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that the covariance spectrum of self-supervised representations act as an implicit prior for downstream linear tasks. It shows that single-penalty ridge on scaled features is equivalent to generalized ridge with direction-specific regularization. It also shows that matching the spectrum to a diagonal task prior recovers the Wiener/Bayes estimator under stated assumptions. Experiments across vision and text encoder tasks study the cost of isotropy, labels required for prior recovery, cross-task transfer and training with non-flat spectrum.

Critical strengths
  • The theory of spectrum as a prior is clean and compelling - the equivalence between single-penalty ridge and generalized ridge provides a useful interpretation of the representation spectrum as directional regularization.
  • The paper meaningfully extends the theory beyond core equivalence - the label complexity analysis and rank characterization provide additional theoretical predictions that are tested empirically.
Critical weaknesses
  • The paper uses post-hoc whitening which is overstated by claiming that isotropy erases information. For positive retained eigenvalues, whitening is an invertible transform. Hence, what changes is the interaction between feature scaling, finite labeled data and a restricted ℓ2\ell_2 regularizer. So, the experiments are mainly demonstrating the remova of a useful regularization/scaling prior but not necessarily removal of information from the representation. It is unclear if the result would hold for sufficiently flexible downstream estimators.
Questions for authors
  • What evidence distinguishes a genuinely learned downstream task prior from generic spectral anisotropy?
  • Were the power law exponent and PC-order vs. prior order choices made using test accuracy? Will the results still hold if these choices are selected using a validation split and isotropy costs computed on a separate test set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.