- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Entropy-Based Regularization (EBReg), which prevents representational collapse in self-supervised learning by maximizing representation entropy. EBReg combines view alignment with Gaussian-smoothed marginal entropy maximization and learns the latent score via denoising score matching, avoiding explicit density estimation. The authors further analyze EBReg’s dynamics in low-variance directions and connect different scale constraints to existing anti-collapse regularizers. The learned score model is also used for test-time adaptation, guiding corrupted representations toward high-density regions of the source latent distribution. Experiments show competitive linear-probe performance on ImageNet-1K and improved robustness on ImageNet-C, at the cost of additional training overhead.
Critical strengths
-
EBReg learns the latent score via denoising score matching and uses it as an entropy-promoting signal to mitigate representational collapse, resulting in a simple and coherent framework. The learned score model can also be reused for score-guided test-time adaptation, serving both pretraining and test-time adaptation.
-
The paper analyzes how EBReg amplifies low-variance directions and consistently observes higher effective rank across different backbones. The theoretical analysis aligns well with the empirical findings and helps explain how EBReg mitigates dimensional collapse.
Critical weaknesses
-
My main concern is the substantial overlap between EBReg and MVEB [1]. MVEB already performs self-supervised learning through view alignment and differential entropy maximization, using the score to construct the entropy gradient. Both methods also implement the entropy term as an inner product between a detached score and the latent representation. Without a discussion or direct comparison with MVEB, I remain concerned about the novelty of the core method.
-
The current experiments do not clearly establish the key advantage of the parameterized denoising score model. There is no direct comparison with MVEB, the linear-probe accuracy gains are generally modest, and training time increases by approximately 13.6%. Score-guided test-time adaptation is also evaluated only by comparing performance before and after adaptation, without standard TTA baselines. It is therefore unclear whether the gains justify the additional complexity or which component is primarily responsible for the improvements.
[1] Wen et al., MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck, IEEE TPAMI, 2024. https://arxiv.org/abs/2403.19078
Questions for authors
-
MVEB [1] also performs self-supervised learning through view alignment and differential entropy maximization, using the score to construct the entropy gradient. Beyond Gaussian smoothing and learning a parameterized score model via denoising score matching, what are the substantive differences between EBReg and MVEB in their objectives and training mechanisms? Are their entropy-gradient updates mathematically equivalent?
-
Could the authors directly compare EBReg with MVEB under the same backbone, augmentations, and training setup, reporting linear-probe accuracy, effective rank, and computational cost? An ablation comparing the Stein gradient estimator with the learned score model would also be helpful. If the authors can clearly explain the substantive differences and demonstrate a clear advantage of the learned score model, I would be willing to revise my score upward.
-
For score-guided test-time adaptation, could the authors compare EBReg with standard TTA methods such as Tent using the same number of adaptation steps and the same set of trainable parameters? Please also clarify how much of the improvement comes from the learned score rather than standard test-time updates to the normalization parameters.
[1] Wen et al., MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck, IEEE TPAMI, 2024. https://arxiv.org/abs/2403.19078
2 comments
The author has solved my problem, and I am willing to raise my score. However, I am not very familiar with this field, so my confidence level is 1.
Response
We thank the reviewer for their valuable feedback and for drawing our attention to MVEB. We address your concerns below.
1. Comparison with MVEB
The main distinctions between EBReg and MVEB are as follows:
a) Parameterized versus kernel-based score estimation. MVEB uses a nonparametric Stein gradient estimator computed from minibatch samples, whereas EBReg learns a parameterized noise-prediction network across successive minibatches. The network can retain information from earlier batches, rather than reconstructing the estimate solely from the current batch. Thus, in MVEB entropy maximization force is estimated from a mini-batch, while EBReg uses a repulsive force from the entire training distribution
b) Entropy objective. EBReg targets the smoothed entropy H(z + σε) for a fixed σ > 0, whereas MVEB targets the entropy H(z) of the clean embeddings and estimates its gradient with a kernel-based score. In EBReg, Gaussian smoothing is part of the objective itself: it introduces a controlled noise scale and produces a smooth, full-dimensional density. This differs from using a bandwidth to regularise a kernel estimator. Consequently, the objectives and their gradients are generally different at positive noise levels.
c) Test-time adaptation. EBReg’s learned score network provides a reusable approximation to the Gaussian-smoothed source-distribution score. With this network frozen, we update only the encoder’s normalization parameters, without labels or retained source examples. A kernel-based implementation of the same source-guided adaptation would require source reference embeddings or another source-distribution approximation.
2. Controlled comparison with MVEB
Following the reviewer’s suggestion, we ran MVEB using the same augmentations, backbone, and training protocol as EBReg. MVEB achieves 67.38% linear-probe accuracy, compared with 67.84 ± 0.11% for EBReg, a difference of 0.46 percentage points. Its effective rank is 1332, approximately 100 units lower than EBReg’s. Under identical hardware and setup (4× A100, batch size 512), MVEB takes 1,330 s/epoch vs 1,220 s/epoch for EBReg (MVEB is 9% slower), with the same peak memory (~12.4 GB/GPU)
3. Matched TTA comparisons and the contribution of the learned score
To assess the contribution of the learned source-distribution score, we added comparisons with Tent and a randomly initialised score network on ViT-H. The results demonstrate that our score-based adaptation achieves better test-time adaptation performance under identical experimental settings. This improvement may be attributed to the more informative adaptation signal provided by the score network. Replacing it with a random score network substantially reduces these gains, supporting this explanation.
All methods use the same evaluation protocol and adaptation settings.
Adaptation protocol. We adapt only the LayerNorm affine parameters of the ViT-H/14 encoder, starting from their pretrained values. All other encoder weights, the linear probe, and the score network remain frozen. We take 10 adaptation steps with learning rate 10−3, β1=0.9, β2=0.999, and no weight decay.
Our method achieves 62.85% mean accuracy across all 15 ImageNet-C corruptions, compared with 60.77% for Tent and 58.19% without adaptation. It therefore exceeds Tent by 2.08 percentage points under the matched protocol.
Replacing the learned score network with a randomly initialised network reduces mean accuracy to 57.50%, below the unadapted baseline. This ablation supports the importance of learning the source-distribution score. Together, these results support the practical value of reusing EBReg’s learned score for test-time adaptation.
ImageNet-C accuracy (%) on ViT-H before and after 10 adaptation steps. The seven corruptions follow Table 5;Bold indicates the highest accuracy in each row.