acceptodds
Under review as a conference paper at ICLR 2027

A Matter of Direction: Eigenframe Distribution Matching for Self-Supervised Learning

Abstract

Self-supervised objectives seek representations that preserve informative variation while avoiding collapse. Yet a representation can distinguish images while losing variation along some directions, a partial form of collapse. Why might such an objective respond weakly to this loss? We investigate this question in methods that match feature distributions through random one-dimensional projections. Each projection mixes many directions, allowing variation in healthy directions to mask a deficiency in another. In a local Gaussian model, we show that this effect becomes stronger as the representation dimension grows. Averaging more projections reduces sampling noise but does not strengthen the expected response. We call this effect spectral dilution. This observation motivates VARTE, a self-supervised objective that replaces random directions with the covariance eigenvectors of features averaged across augmented views. These directions separate the representation's modes of variation, allowing the matching objective to examine each one directly. We analyze this choice theoretically and test its consequences through controlled interventions and image representation learning. On ImageNet-1K, VARTE reaches 74.7% linear-probe accuracy with a ViT-B/16 and transfers competitively across recognition and specialized-domain tasks. Our work highlights a simple principle: preventing collapse depends not only on the distribution we ask features to match, but also on the directions along which we measure them.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, dimensional collapse, distribution matching, optimal transport, covariance geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fSkBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper can be regarded as a follow-up to LeJEPA. It points out that the random projection directions adopted by LeJEPA may not sufficiently capture subtler collapse. The authors adopt spectral directions to replace the original choice in LeJEPA, and further support their claim through both experimental and theoretical analyses.

Critical strengths

The design is well-motivated and the whole paper is easy to follow. The supporting evidence, including Theorem 1 and Figure 3, explains and verifies how random projections dilute localized variance errors. The authors also provide a theoretical justification for their choice: Theorem 2 shows that the covariance eigenframe maximizes the average scale discrepancy among complete orthogonal frames. They also conduct targeted ablations to directly examine the contribution of the eigenframe.

Critical weaknesses

The mechanistic explanation of the downstream benefit remains incomplete. I have a concern about the motivation: preserving variation across all representation directions does not by itself explain why downstream tasks should benefit. Is there any support to connect them? I know section D.4 provides useful backbone diagnostics. However, the training constraint is applied after the projector, leaving an indirect connection between the proposed regularization and the geometry and task-relevant information retained by the backbone.

The contribution of view averaging is not independently evaluated. section B.3 explains the decomposition, but the direction ablations retain view averaging and Table S4 examines other design choices. A controlled comparison with per-view matching would help clarify its contribution. The use of view averaging is reasonably motivated, but its contribution has not been isolated through a dedicated ablation.

Questions for authors

Could the authors disentangle the computational savings due to view averaging and the smaller projection budget from the additional overhead of covariance construction and eigendecomposition? The overall costs are reported, but the contributions of these components remain unclear. This is just a suggestion to make this paper more soild.

Overall, I think this paper is well-motivated. It has provided me with new insights and understanding, and the improvements demonstrate the effectiveness of their design. Hence, I have a positive view of this work, even though the connection between their design choices and downstream performance remains indirect.

1 comment
Reviewer_fSkBOpenReview ↗

Final Assessment

Thank you for your response. I'm very familiar with this topic, and I believe there are something new from the paper, therefore I keep my positive assessmemt. Even though this is only a minor refinement, I don't think ICLR only accepts papers that fundamentally change how the field thinks about a problem.

Reviewer_mcHWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effect of projection directions in self-supervised learning. The proposed VARTE matches view-averaged features to fixed Gaussian quantiles along their empirical covariance eigenvectors, while encouraging agreement between augmented views. Experimental results show that the proposed method achieves competitive performance.

Critical strengths
  1. This paper presents an interesting perspective of direction in self-supervised learning.

  2. The experiments are comprehensive.

  3. This paper is easy to follow with clear writing.

Critical weaknesses

With ViT-B/16, VARTE underperforms VISReg on ImageNet-1K linear probing and other downstream tasks in Tab. 1-4. This shows the benefits appear inconsistent. It remains unclear when improved weak-mode preservation leads to better task-relevant representations.

Questions for authors
  1. Is it necessary to preserve all weak directions? Do these directions contain additional information for downstream tasks, or is retaining the existing non-collapsed subspace sufficient? A comparison across retained dimensions could clarify whether preserving these weak modes is necessary for downstream performance.

  2. What motivates estimating the eigenframe from view-averaged features? Could the authors compare this choice with estimates based on global-view averages or local-view averages? My concern is whether local crops affect the stability and usefulness of the estimated directions.

Reviewer_pYMBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of partial dimensional collapse in self-supervised learning when matching feature distributions using random observation directions. Such collapse can be difficult to detect because random directions mix information from multiple feature dimensions, making the loss of variance along an individual dimension less apparent. The authors refer to this phenomenon as spectral dilution. Under a local perturbation model of the population covariance matrix, they derive that the response of random observation directions to scale errors in an individual weak covariance mode diminishes as the representation dimensionality increases. The authors further prove that, for a given positive semidefinite covariance matrix, the orthogonal frame formed by its covariance eigenvectors maximizes eigenframe exposure to scale errors. Based on this analysis, the paper proposes VARTE. The method averages the features of multiple augmented views of the same image, constructs an adaptive eigenframe from the covariance matrix of mini-batch features, matches the projected features along each eigenframe direction to a fixed Gaussian target grid, and additionally enforces view agreement across augmented views. The experiments include controlled analyses of variance errors, ablation studies on ImageNette, pretraining with ViT-B/16 and ViT-L/14 on ImageNet-1K, and evaluations on downstream linear probing, specialized-domain transfer, full fine-tuning, and ADE20K semantic segmentation. The results indicate that the proposed method responds more directly to insufficient variation along individual directions.

Critical strengths

Strength 1: The paper identifies a relatively clear, interesting, and meaningful research problem.

Strength 2: The problem is supported by substantial theoretical analysis.

Rather than simply introducing an additional regularization term to address representation collapse, the paper analyzes the directions along which distribution-matching objectives actually observe the features. Starting from a single weak covariance mode, the authors explain how random observation directions mix the variance error along that mode with variations along other modes, and provide a quantitative analysis of how this effect changes with the representation dimensionality.

Critical weaknesses

Weakness 1: The theoretical advantages do not translate sufficiently into competitive representation-learning performance.

The paper's central argument is that distribution matching along eigenframe directions can detect and correct weak modes more directly. However, this mechanistic advantage does not translate into consistent and comprehensive performance gains on the main downstream evaluations. The proposed method does not demonstrate a competitive advantage on any of the downstream tasks reported in the main paper.

Weakness 2: The experimental validation remains incomplete.

The downstream evaluations in the paper leave several important scenarios insufficiently explored. For example, the evaluation lacks common downstream settings such as training on larger-scale datasets, learning from low-quality data, and image generation guidance. VISReg may serve as a useful reference for considering additional evaluation settings.

Questions for authors

Question 1: Why does stronger correction of weak modes not lead to better overall representation quality?

The paper presents a high-quality theoretical analysis and an insightful central idea. However, the connection between mitigating spectral dilution and learning more discriminative and transferable representations remains unclear. The authors also acknowledge that the proposed strategy does not yield reliable downstream performance gains. What practical significance, then, does improving the response to individual spectral modes have? The authors should further explain how this mechanistic improvement translates into better representation quality and provide more comprehensive experimental evidence to validate its practical value.

Question 2: Could the method be evaluated on larger-scale datasets and compared with methods designed for large-scale pretraining, such as more recent versions of DINO, to assess its scalability?

Question 3: The core idea of this paper is promising. Would it be useful to transfer the proposed strategy of using eigenframe projections to mitigate spectral dilution into existing methods based on random observation directions, and evaluate whether it improves their performance? Such experiments could help validate the effectiveness of the proposed approach.

Question 4: The view-agreement term is designed in a relatively simple manner. Could this design be a contributing factor to the method's limited performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.