acceptodds
Withdrawn from ICLR 2027

AdaDim: Dimensionality Adaptation for SSL Representational Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A key factor in effective Self-Supervised Learning (SSL) is preventing dimensional collapse, where higher-dimensional representation spaces span a lower-dimensional subspace. Therefore, SSL optimization strategies involve guiding a model to produce a representation space with a higher dimensionality through objectives that encourage feature decorrelation or sample uniformity. A higher dimensionality indicates that the representation space has greater feature diversity which is useful for generalization to downstream tasks. In addition to dimensionality optimization, SSL algorithms also utilize a projection head that maps the representation space to an embedding space where the optimization objective is applied. Recent work has characterized the projection head as a filter of noisy or irrelevant features by reducing the mutual information between the representation and embedding spaces. Therefore, the current literature views a good SSL representation space as one with a high dimensionality and a low mutual information between each respective space. However, we directly oppose this perspective and instead assert that the best performing SSL model arrives at an ideal balance between both dimensionality and mutual information. This conclusion is supported through a novel training dynamics analysis that reveals that increases in dimensionality due to feature decorrelation at the start of training lead to correspondingly higher mutual information, while increases in dimensionality due to samples distributing uniformly in a high-dimensional space at the end of training cause mutual information to plateau or decrease. To leverage these dynamics, we introduce AdaDim, a method that adaptively balances between dimensionality and mutual information by identifying situations where both families of SSL approaches are better utilized. We show that AdaDim results in performance exceeding common SSL baselines by as much as 3% accuracy on common SSL benchmark tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self Supervised Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TZAAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes AdaDim, a self-supervised learning framework that adaptively combines the SimCLR and VICReg losses based on the effective rank of the embedding space. Additionally, they introduce a regularization term that encourages higher mutual information between the encoder representation space RR and projection head embedding space ZZ. The paper's main claim is that maximizing representation dimensionality H(R)H(R) or minimizing mutual information I(R;Z)I(R;Z) independently is suboptimal. Instead, the authors argue that downstream performance benefits from balancing these two quantities.

Critical strengths

None

Critical weaknesses
  1. Presentation and clarity. The paper suffers from several presentation issues that make it difficult to follow. References and citations are inconsistently formatted throughout the manuscript, and several figures lack proper subfigure captions and labeling. Furthermore, the methodology section does not explicitly define the individual loss functions used in the proposed framework, instead referring the reader to the appendix. This makes it difficult to understand the proposed method from the main paper alone.
  2. Fairness and validity of experimental comparisons. The empirical evaluation raises several concerns regarding the fairness and validity of the reported comparisons. In Table 4, the authors compare AdaDim against multiple SSL frameworks using ResNet-18. However, the hyperparameter settings deviate from the original training configurations of several baselines, including the use of smaller batch sizes, which can substantially degrade the performance of evaluated baselines. More concerningly, Table 5 reports ImageNet-1K results for ReSA [1] that do not match those reported in the official ReSA manuscript. Specifically, ReSA reports top-1 accuracies of 71.9% and 73.4% for ResNet-50 trained for 100 and 200 epochs, respectively, using a batch size of 256. In contrast, the authors report 70.8% and 71.6% for the corresponding configurations. This discrepancy is substantial and is not adequately explained in the manuscript.

[1] Weng, Xi, et al. "Clustering properties of self-supervised learning.", ICML, 2025.

Questions for authors

I have no questions that could materially affect my assessment.

Reviewer_UkphOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how representation dimensionality and mutual information evolve during SSL training. It proposes AdaDim, which adaptively mixes feature decorrelation and sample uniformity objectives with mutual information regularization. Experiments report gains across natural and medical images.

Critical strengths
  1. The paper raises an interesting question about whether higher dimensionality always means better representations and connects it to a simple adaptive objective.
  2. The experiments cover diverse datasets and architectures. The ImageNet results and component ablations provide useful evidence that AdaDim can improve representation learning.
Critical weaknesses
  1. Scope of the explanation. It remains unclear whether the proposed dynamics also hold for prototype methods and JEPA. Comparing their final accuracy does not establish that they follow the same training phases. The paper would benefit from clarifying where its explanation applies and how these settings fit into it.

  2. Why this balance? The results suggest that intermediate values can work well, but do not establish why this balance should be optimal. Table 1 tests dimensionality alone, while the theoretical upper bound does not guarantee better downstream performance. I would like a clearer explanation of when entropy or mutual information should increase or decrease, and why AdaDim's rule makes the right choice.

  3. What the dynamics show. Figures 14 and 15 already visualize AdaDim's trajectories, but the connection to performance remains unclear. AdaDim stays fairly close to VICReg in Figure 14. Showing how the trajectories and accuracy change when individual components are removed would help explain whether the proposed balance actually accounts for the gains.

  4. Missing related work. The discussion should include DSE [1]. Its metric combines effective dimensionality with class separability to study and mitigate degradation during SSL training. Explaining how AdaDim's perspective differs would sharpen the novelty claim, especially since dimensionality alone is already questioned by this work.

[1] Exploring Structural Degradation in Dense Representations for Self-supervised Learning. NeurIPS 2025.

Questions for authors

Please refer to the weaknesses.

Reviewer_PgDiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures how the dimensionality H(R) (effective rank) of SSL representations and the mutual information I(R;Z) between representation and projector embedding move during training. It argues three things. The best models sit at a balance between high H(R) and low I(R;Z). Early feature decorrelation raises I(R;Z), while late sample uniformity flattens or lowers it. The two loss families should therefore be mixed according to training phase. AdaDim mixes the InfoNCE loss LSUL_{SU} and the VICReg loss LFDL_{FD} through α=ER(Z,Z′)/D\alpha = ER(Z,Z')/D, and adds −βLmut-\beta L_{mut} with β=γα\beta = \gamma\alpha. In the baseline setting without a momentum encoder, AdaDim slightly exceeds the strongest baseline on 6 of 8 datasets. With a momentum encoder, it beats Resa by +3.32 (Cifar100), +3.95 (TinyImageNet), and +1.24 (ImageNet R50@200).

Critical strengths
  • The observation that I(R;Z) moves in stages is novel.
  • AdaDim adds little training cost: 23.48 s/epoch against 22.01 for SimCLR and 75.10 for Resa (Table 13). Its rank update is an SVD on 10 batches every 20 epochs.
Critical weaknesses
  • My main concern is the statistical support for the first claim. The U-test covers only H(R). VICReg's significance rests on one model, at effective rank 795; removing it gives p=.612. On the I(R;Z) axis, no method shows a significant interior optimum (recomputed from Table 14).
  • A second concern is that every gain of more than 3 points over the strongest comparator comes from the momentum setting: TinyImageNet +3.95 and CUB-200 +4.05 vs Resa, Cifar100 +3.20 vs VICReg. Read against the strongest baseline, the abstract's "as much as 3%" also rests on this comparison. The only same-setting controls are Resa and the Table 20 VICReg.
Questions for authors
  • As a clarification, did the VICReg of Table 20 (1000 epochs) use a momentum encoder? Do you have results for VICReg, SimCLR, fixed α\alpha=0.5, and β\beta-free AdaDim with momentum?
  • What supports a balance point in I(R;Z), and how large is run-to-run noise? Significant two-variable optima would make the first claim partly supported.
  • On which split were the tuned γ\gamma values chosen (Table 3, and γ=1e−1\gamma=1e-1 for ImageNet-100 in Table 10)? At fixed γ=1e−4\gamma=1e-4, ImageNet-100 gives 79.30 (batch 128) and 79.64 (batch 256), below Barlow Twins 80.38.
  • As a clarification, is α\alpha computed on R or Z? Section 4 first uses H(R), then ER(Z,Z'), as Algorithm 1 does. Is rank_list in Algorithm 1 reset at each α\alpha update or accumulated? Does Table 2's β=1\beta=1 row mean that β\beta is fixed at γ\gamma?
  • B.2 and Fig. 9(c) say ln⁡∣ΣZ∣\ln|\Sigma_Z| stays constant as cluster variance grows, but under a PCA projection the variance of Z should grow with R's. Was Z whitened, and how large is the diagonal noise? This decides whether the two-component I(R;Z) decrease in Section 3.1 is an artifact.
  • With γ=1e−4\gamma=1e-4, β\beta is at most 1e-4. What share of the total gradient does LmutL_{mut} carry? Fig. 14(a) has no γ=0\gamma=0 point, so it is unclear whether LmutL_{mut} blocks the late drop in I(R;Z) and explains the +0.43 in Table 2. As a suggestion, please report a seeded γ\gamma sweep that includes γ=0\gamma=0.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.