acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Collapse Prevention with Score-Based Entropy Maximization

Abstract

Diffusion models have demonstrated the effectiveness of score-based objectives for generative modeling, yet their potential for self-supervised representation learning remains underexplored. We introduce Entropy-Based Regularization (EBReg), a framework that uses latent score estimation to regularize the latent distribution during self-supervised training. EBReg jointly learns an encoder and a latent score model, and utilizes the learned score to provide a diversity-preserving signal that prevents representational collapse. Motivated by a mutual-information lower bound, we formulate the objective as view alignment with smoothed marginal entropy maximization and optimize the marginal entropy without explicit density evaluation. The same score model can additionally be used at test time to refine corrupted representations toward the training latent distribution. Empirically, EBReg learns competitive representations on ImageNet-1K, achieving 75.57% linear-probe accuracy with ViT-L. Across diverse architectures, score-guided test-time adaptation further improves robustness on ImageNet-C, yielding an average relative accuracy gain of approximately 5%.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self Supervised Learning, Joint Embedding Predictive Architecture

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1PgmOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Entropy-Based Regularization (EBReg), which prevents representational collapse in self-supervised learning by maximizing representation entropy. EBReg combines view alignment with Gaussian-smoothed marginal entropy maximization and learns the latent score via denoising score matching, avoiding explicit density estimation. The authors further analyze EBReg’s dynamics in low-variance directions and connect different scale constraints to existing anti-collapse regularizers. The learned score model is also used for test-time adaptation, guiding corrupted representations toward high-density regions of the source latent distribution. Experiments show competitive linear-probe performance on ImageNet-1K and improved robustness on ImageNet-C, at the cost of additional training overhead.

Critical strengths
  1. EBReg learns the latent score via denoising score matching and uses it as an entropy-promoting signal to mitigate representational collapse, resulting in a simple and coherent framework. The learned score model can also be reused for score-guided test-time adaptation, serving both pretraining and test-time adaptation.

  2. The paper analyzes how EBReg amplifies low-variance directions and consistently observes higher effective rank across different backbones. The theoretical analysis aligns well with the empirical findings and helps explain how EBReg mitigates dimensional collapse.

Critical weaknesses
  1. My main concern is the substantial overlap between EBReg and MVEB [1]. MVEB already performs self-supervised learning through view alignment and differential entropy maximization, using the score to construct the entropy gradient. Both methods also implement the entropy term as an inner product between a detached score and the latent representation. Without a discussion or direct comparison with MVEB, I remain concerned about the novelty of the core method.

  2. The current experiments do not clearly establish the key advantage of the parameterized denoising score model. There is no direct comparison with MVEB, the linear-probe accuracy gains are generally modest, and training time increases by approximately 13.6%. Score-guided test-time adaptation is also evaluated only by comparing performance before and after adaptation, without standard TTA baselines. It is therefore unclear whether the gains justify the additional complexity or which component is primarily responsible for the improvements.

[1] Wen et al., MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck, IEEE TPAMI, 2024. https://arxiv.org/abs/2403.19078

Questions for authors
  1. MVEB [1] also performs self-supervised learning through view alignment and differential entropy maximization, using the score to construct the entropy gradient. Beyond Gaussian smoothing and learning a parameterized score model via denoising score matching, what are the substantive differences between EBReg and MVEB in their objectives and training mechanisms? Are their entropy-gradient updates mathematically equivalent?

  2. Could the authors directly compare EBReg with MVEB under the same backbone, augmentations, and training setup, reporting linear-probe accuracy, effective rank, and computational cost? An ablation comparing the Stein gradient estimator with the learned score model would also be helpful. If the authors can clearly explain the substantive differences and demonstrate a clear advantage of the learned score model, I would be willing to revise my score upward.

  3. For score-guided test-time adaptation, could the authors compare EBReg with standard TTA methods such as Tent using the same number of adaptation steps and the same set of trainable parameters? Please also clarify how much of the improvement comes from the learned score rather than standard test-time updates to the normalization parameters.

[1] Wen et al., MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck, IEEE TPAMI, 2024. https://arxiv.org/abs/2403.19078

2 comments

We thank the reviewer for their valuable feedback and for drawing our attention to MVEB. We address your concerns below.

1. Comparison with MVEB

The main distinctions between EBReg and MVEB are as follows:

a) Parameterized versus kernel-based score estimation. MVEB uses a nonparametric Stein gradient estimator computed from minibatch samples, whereas EBReg learns a parameterized noise-prediction network across successive minibatches. The network can retain information from earlier batches, rather than reconstructing the estimate solely from the current batch. Thus, in MVEB entropy maximization force is estimated from a mini-batch, while EBReg uses a repulsive force from the entire training distribution

b) Entropy objective. EBReg targets the smoothed entropy H(z + σε) for a fixed σ > 0, whereas MVEB targets the entropy H(z) of the clean embeddings and estimates its gradient with a kernel-based score. In EBReg, Gaussian smoothing is part of the objective itself: it introduces a controlled noise scale and produces a smooth, full-dimensional density. This differs from using a bandwidth to regularise a kernel estimator. Consequently, the objectives and their gradients are generally different at positive noise levels.

c) Test-time adaptation. EBReg’s learned score network provides a reusable approximation to the Gaussian-smoothed source-distribution score. With this network frozen, we update only the encoder’s normalization parameters, without labels or retained source examples. A kernel-based implementation of the same source-guided adaptation would require source reference embeddings or another source-distribution approximation.

2. Controlled comparison with MVEB

Following the reviewer’s suggestion, we ran MVEB using the same augmentations, backbone, and training protocol as EBReg. MVEB achieves 67.38% linear-probe accuracy, compared with 67.84 ± 0.11% for EBReg, a difference of 0.46 percentage points. Its effective rank is 1332, approximately 100 units lower than EBReg’s. Under identical hardware and setup (4× A100, batch size 512), MVEB takes 1,330 s/epoch vs 1,220 s/epoch for EBReg (MVEB is 9% slower), with the same peak memory (~12.4 GB/GPU)

3. Matched TTA comparisons and the contribution of the learned score

To assess the contribution of the learned source-distribution score, we added comparisons with Tent and a randomly initialised score network on ViT-H. The results demonstrate that our score-based adaptation achieves better test-time adaptation performance under identical experimental settings. This improvement may be attributed to the more informative adaptation signal provided by the score network. Replacing it with a random score network substantially reduces these gains, supporting this explanation.

All methods use the same evaluation protocol and adaptation settings.

Adaptation protocol. We adapt only the LayerNorm affine parameters of the ViT-H/14 encoder, starting from their pretrained values. All other encoder weights, the linear probe, and the score network remain frozen. We take 10 adaptation steps with learning rate 10−310^{-3}, β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, and no weight decay.

Our method achieves 62.85% mean accuracy across all 15 ImageNet-C corruptions, compared with 60.77% for Tent and 58.19% without adaptation. It therefore exceeds Tent by 2.08 percentage points under the matched protocol.

Replacing the learned score network with a randomly initialised network reduces mean accuracy to 57.50%, below the unadapted baseline. This ablation supports the importance of learning the source-distribution score. Together, these results support the practical value of reusing EBReg’s learned score for test-time adaptation.

ImageNet-C accuracy (%) on ViT-H before and after 10 adaptation steps. The seven corruptions follow Table 5;Bold indicates the highest accuracy in each row.

CorruptionBeforeRandom score (10 steps)Tent (10 steps)Ours (10 steps)
Fog58.4157.2260.9763.08
Impulse noise51.5050.6756.3259.16
Motion blur58.3258.0460.4363.17
Glass blur45.2343.9347.7852.81
Zoom blur51.1950.5853.9456.70
Gaussian noise50.0749.5055.9859.65
Contrast60.7159.4663.4365.39
Mean (15 corruptions)58.1957.5060.7762.85

The author has solved my problem, and I am willing to raise my score. However, I am not very familiar with this field, so my confidence level is 1.

Reviewer_vzNfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a method of regularizing self-supervised feature learning using a jointly learned diffusion score estimate of the data distribution to avoid feature collapse. The authors prove their method is a surrogate for a mutual information lower bound and consequently should avoid feature collapse given a well trained, or network of sufficient capacity, score estimator. The linear probe classification accuracy performance of this method is shown to be comparable to relevant baselines on ImageNet-1k and a test-time adaptation experiment on ImageNet-C.

Critical strengths
  • The authors provide a clearly written description of their method that is well motivated. The methodology is shown to be theoretically supported by derivation of a mutual information lower bound and supporting proofs.

  • Over the chosen experiments, the authors method performs broadly on par with related methods. The authors are transparent about the additional computational cost their method requires over SIGReg, which is marginal, and given the provided guarantees for avoiding representation collapse acceptable.

Critical weaknesses
  • Beginning with some minor style motes. The notation for mutual information I(h(1);h(2))I(h^{(1)}; h^{(2)}) is not explicitly introduced in the paragraph following eq(1) in the text. I would suggest the authors add a few words stating this is the notation for mutual information, as they did for the differential entropy preceding this text. In the same paragraph, the authors write "Since h~(1)\tilde{h}^{(1)} is obtained from h(1)h^{(1)} by adding independent Gaussian noise, h(2)→h(1)→h~(1)h^{(2)}\to h^{(1)}\to \tilde{h}^{(1)} forms a Markov chain.". The construction of this Markov chain at first glance was somewhat confusing as it implies there is a transition probability between h(2)→h(1)h^{(2)}\to h^{(1)}, which is not clear from how the transformations t,t′∈Tt,t'\in \mathcal{T} are selected. This is not clarified till the authors subsequently state this more explicitly and chose qτ(⋅∣h(2))q_\tau(\cdot \mid h^{(2)}). It would benefit readers if the authors improved the presentation of this portion of the text accordingly.

  • In Section.3.1 the authors mention the sensitivity of their method, based on eq(4), to the scale of the latent features; line 231 to line 233. I took the sentence "Increasing latent scale can [...]" to mean "The norm of the latent features [...]" however the writing is not very precise and should be edited to describe mathematically what exactly the authors mean. Moreover, the authors mention discussion of how this is avoided by use of a normalizing constant, but the included reference is not correct, and I can't locate the relevant section of text within the appendix. The inclusion of this portion of the text needs to be addressed; moreover, the importance of the normalization method chosen deserves mention in the provided ablations in the experimental section.

  • My final comment is about the style template. The authors opted to include the required AI use statement within the appendix. I believe for the current 2027 policy this statement needs to follow the Reproducibility Statement before the References. In 2026 the policy stated it could appear in the appendix, but I'm not sure this is the case for the current review cycle.

Questions for authors

Seem comments on critical weaknesses.

  • I have one question that I felt did not fit in with those above. It is my understanding that feature learning (or representation learning), at least in part, is motivated by trying to find compact latent representations that are useful for downstream applications (e.g., classification) mirroring the use of VAEs in latent space image models. Under this view, why do the authors opt to preferentially score high effective rank representations as opposed to lower ones (assuming no representation collapse)?

I am willing to improve my score based on the authors responses to my comments on critical weaknesses.

2 comments

We thank the reviewer for recognising the clarity and theoretical motivation of our approach. Your suggestions identify several places where the presentation can be more precise. Below, we clarify them.

1. Clarifying mutual information and the Markov chain

We have now explicitly introduced I(⋅;⋅)I(\cdot;\cdot) as mutual information after Equation (1) and clarified the construction of the Markov chain. We generate the perturbed embedding as

\qquad \epsilon\sim\mathcal N(0,I),$$ where $\epsilon$ is independent of both $h^{(1)}$ and $h^{(2)}$. Consequently, $$p(\tilde h^{(1)}\mid h^{(1)},h^{(2)}) =p(\tilde h^{(1)}\mid h^{(1)}) =\mathcal N(\tilde h^{(1)};h^{(1)},\sigma^2I).$$ Thus, $\tilde h^{(1)}$ is conditionally independent of $h^{(2)}$ given $h^{(1)}$, establishing the Markov chain $$h^{(2)}\longrightarrow h^{(1)}\longrightarrow\tilde h^{(1)}.$$ The data-processing inequality therefore gives $I(h^{(2)};\tilde h^{(1)})\le I(h^{(2)};h^{(1)})$. We have revised the manuscript to state this explicitly and distinguish it from the subsequent choice of the variational density $q_\tau(\tilde h^{(1)}\mid h^{(2)})$, which is used to derive the alignment term and is not needed to establish the Markov chain. ## 2. Clarifying latent scale and embedding normalization By “increasing latent scale,” we mean rescaling the embeddings as $h\mapsto ah$, with $a>1$. This increases their norms without adding information and can increase the entropy term, although it also affects the alignment term. We control this scale by applying $\ell_2$ normalization to the projector output $u$: $$h=\frac{u}{\|u\|_2}.$$ Thus, every clean embedding satisfies $\|h\|_2=1$, and multiplying $u$ by a positive scalar leaves the normalized embedding unchanged. This is an explicit constraint on the representations, rather than a normalizing constant in the objective. Under this spherical constraint, the uniform distribution maximises the smoothed entropy, as established by the symmetry-and-concavity argument in Appendix E. We will revise Section 3.1 to state the normalization operation explicitly and replace the broad reference to Section 3.1 with a direct reference to Appendix E(ii) and its smoothing discussion. ## 3. Placement of the AI-use statement We have moved the AI-use statement to be immediately after the Reproducibility Statement. ## 4. Interpreting effective rank RankMe (Garrido et al., 2023) identifies effective rank as a useful unsupervised indicator of downstream performance in joint-embedding SSL. Our objective is to improve the information retained by the representation while keeping its dimensionality fixed, consistent with the motivation of redundancy-reduction and collapse-prevention methods such as Barlow Twins (Zbontar et al., 2021) and VICReg (Bardes et al., 2022). However, RankMe also acknowledges that rank is not the only factor affecting performance. We therefore assess effective rank alongside downstream accuracy and have clarified that higher effective rank alone does not guarantee better downstream performance.

I have read the responses and the revised version of the text.

My main criticisms of the text have been addressed and generally the text is clearer. I have also looked at the review supplied by LPdx and agree the text time adaptation of the text would benefit form the inclusion of the table the authors supplied in response which is currently not included in the revised text. The additional comments added by the authors to Appx.E also helps in clarifying some of the claims and overall the material seems correct under typical assumptions. There remains a notational issues in Appx.E with eq(17) I believe the wrong latex command for an indicator was used.

Overall my opinion of the paper remains essentially the same as before; the method is nice and appears decently performant. As raised by other reviewers there remains some questions about how clearly distinguished the work is from related works and the text time adaption section could benefit from the inclusion of additional experiments. At the moment I will be keeping my original score for these reasons.

Reviewer_LPdxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Entropy-Based Regularization (EBReg), a novel self-supervised representation learning framework designed to prevent representational collapse without relying on negative pairs or fixed distribution matching. Motivated by a variational mutual information lower bound, EBReg combines view alignment with smoothed marginal entropy maximization, where the marginal entropy gradient is estimated dynamically via a learned latent score network trained by denoising score matching. Additionally, the learned score model can be naturally repurposed at test time to guide corrupted representations toward the source latent distribution by updating only the encoder’s normalization parameters. Experiments on ImageNet-1K with convolutional and Vision Transformer architectures demonstrate that EBReg achieves competitive linear probing performance, and score-guided test-time adaptation delivers consistent robustness improvements on ImageNet-C.

Critical strengths
  • By combining a lower bound on mutual information—comprising alignment and marginal entropy—with the score-matching objective used in diffusion models, this approach employs a smoothed estimate of marginal entropy. This cleverly sidesteps the issue of undefined entropy that arises from degenerate measures in high-dimensional latent spaces. Furthermore, it eliminates the need for explicit density estimation or contrastive learning with negative samples, resulting in a theoretically sound framework with a clear, well-motivated foundation.
  • The learned latent-space score network serves a dual purpose: during self-supervised pre-training, it acts as a dynamic anti-collapse regularizer (by providing a repulsive force field), while during test-time adaptation, it can be directly reused to enhance robustness against input corruptions—a design that is both innovative and practically valuable.
  • The paper validates its effectiveness across several mainstream backbone networks (ResNet-50, ViT-L, and ViT-H), achieving performance in linear probing comparable to that of leading non-contrastive methods, while also demonstrating consistent robustness gains on ImageNet-C (an average improvement of approximately 5%).
Critical weaknesses
  • The performance improvements are relatively small, and their reliability is not evaluated

Although EBReg achieves competitive results, the improvements over existing methods are quite limited. As shown in Table 2, EBReg achieves 67.92% accuracy on ResNet-50, slightly below VICReg's 68.02%. On ViT-L, the improvement over SIGReg is only 0.25 percentage points. On ViT-H, adding EBReg improves accuracy from 79.33% to 79.41%. Meanwhile, Table 4 shows that EBReg increases the training time per epoch by around 13.6%.

The paper also does not report results from multiple runs or provide standard deviations. Given how small some of these improvements are, it is difficult to tell whether they are consistent and whether they justify the additional training cost.

  • The TTA experiments lack comparisons with existing methods Score-guided TTA is presented as one of the main contributions. However, Table 5 only reports the performance before and after applying the proposed adaptation method. There are no direct comparisons with existing TTA approaches, such as Tent or simple normalization-statistics adaptation.

The current results show that score-guided TTA can improve performance, but they do not tell us whether it works better than existing alternatives. Since TTA is an important part of the paper, I think comparisons with standard TTA baselines are necessary to better support this contribution.

  • The discussion of smoothed entropy in Appendix E is not fully rigorous

In the "Effect of the smoothing" section of Appendix E, the authors state that adding noise increases entropy and that the smoothed entropy converges to the original entropy as the noise level approaches zero. However, this statement does not directly apply to distributions supported on spheres or discrete codes, where the entropy is defined differently from the differential entropy of the smoothed distribution. While this may not invalidate the conclusions about uniform distributions being optimal, the statement itself needs clarification and correction.

Questions for authors

The questions are all above.

4 comments

We thank the reviewer for recognising the role of Gaussian smoothing in defining the entropy objective and the value of reusing the learned score for test-time adaptation. We address your concerns about experimental reliability and TTA adaptation baselines below.

1. Small performance differences

We agree that the reported differences do not establish a consistent accuracy advantage over existing methods. In particular, EBReg is slightly below VICReg on ResNet-50, and the gains on ViT-L and ViT-H are small, but with the additional benefit of test-time adaptation. We will revise the discussion to emphasise competitive performance rather than consistent superiority.

Our central contribution is a score-based mechanism for optimising smoothed marginal entropy without negative pairs or an explicit density model. The experiments assess whether this mechanism can prevent collapse while learning competitive representations. The same learned score network also provides a source-distribution signal for test-time adaptation.

2. The paper also does not report results from multiple runs or provide standard deviations.

BackboneNumber of seedsLinear-probe accuracy, mean ± sample SD
ResNet-50367.84 ± 0.11 %

We report the mean accuracy and sample standard deviation across three ResNet-50 seeds to demonstrate that EBReg is robust to the choice of seed and exhibits less variance across initialisation, randomness, etc. We would include the standard deviation of the proposed method in all three settings in the final version, as currently we have computational constraints.

We note that reporting multiple seeds is not a common practice in SSL (e.g., LeJEPA, I-JEPA) because training jobs are expensive and even small changes on ImageNet scale are statistically significant.

3. Comparisons with existing test-time adaptation methods

Following your comment, we added comparisons with TENT using ViT-H. These results demonstrate that the score-guided test-time adaptation provides a better adaptation signal than TENT, resulting in improved TTA performance. These additional evaluations use a newly trained linear probe, which accounts for the difference in unadapted accuracy from Table 5 of the manuscript.

ImageNet-C accuracy (%) on ViT-H before and after 10 adaptation steps. The seven corruptions follow Table 5; the final row averages all 15 corruptions. Bold indicates the highest accuracy in each row.

CorruptionBeforeTent (10 steps)Ours (10 steps)
Fog58.4160.9763.08
Impulse noise51.5056.3259.16
Motion blur58.3260.4363.17
Glass blur45.2347.7852.81
Zoom blur51.1953.9456.70
Gaussian noise50.0755.9859.65
Contrast60.7163.4365.39
Mean (15 corruptions)58.1960.7762.85

4. The discussion of smoothed entropy in Appendix E is not fully rigorous

We thank the reviewer for identifying this error. We agree that the original statement incorrectly compared entropy defined relative to the surface measure on the sphere or the counting measure for discrete codes with the ambient differential entropy after Gaussian smoothing. We have removed the claim that smoothing necessarily increases these entropies and that the smoothed entropy converges to them as the noise vanishes.

The uniformity conclusions remain valid through a separate symmetry-and-concavity argument. Rotating a distribution on the sphere or permuting the coordinates of the one-hot codes leaves its smoothed entropy unchanged because the Gaussian noise is isotropic. Averaging over these transformations produces the corresponding uniform distribution, and concavity of differential entropy ensures that this averaging cannot decrease entropy. Thus, the uniform distributions remain maximisers of the smoothed entropy for every positive noise level. As the reviewer notes, this issue does not necessarily invalidate the uniformity conclusions; we make this correction in Appendix E.

Dear Reviewer LPdx,

We hope you have had a chance to consider our response and would appreciate any further feedback. We would be happy to clarify any remaining questions or concerns.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Consistency and Reliability of EBReg

Statistical reliability and computational cost. We conducted additional experiments on CIFAR-10 with ResNet-18 using five random seeds per method. Both EBReg and VICReg use the EBJEPA[1] library with identical hardware, augmentations, and training schedules.

EBReg achieves 90.48 ± 0.13% linear-probe accuracy, compared with 89.68 ± 0.16% for VICReg—a gain of 0.80 percentage points, with slightly lower variability across seeds. This improvement incurs a 5.7% increase in steady-state time per epoch and an additional 2.0 minutes of total training time (6.2%). These results support the robustness of EBReg to random initialization while quantifying its modest computational overhead.

MethodLinear-probe accuracy (%)Runtime (min)Time/epoch (s)
VICReg89.68 ± 0.1632.2 ± 0.46.36 ± 0.08
EBReg (ours)90.48 ± 0.1334.2 ± 0.26.72 ± 0.04

*Values are mean ± sample standard deviation over five random seeds. We have also added training curves to the revised manuscript (Section 4.2, Figure 3), showing the evolution of linear-probe accuracy and effective rank across training epochs. The smooth trajectories suggest stable training dynamics, while the small across-seed standard deviations indicate consistent performance across runs.

[1] A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures, Basile Terver et.al

AuthorsOpenReview ↗

Results on ViT-L for 3 random seeds

We trained ViT-L on ImageNet-1K using our method with three different random seeds. Across three training seeds on ImageNet-1K, ViT-L trained with EBReg achieved a mean linear-probe accuracy of 75.56%, with a standard deviation of 0.03 percentage points. This mean exceeded the reported SIGReg baseline accuracy by 0.24 percentage points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.