- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ScoreCLR, a self-supervised learning method that replaces contrastive repulsion with an estimated gradient of latent entropy, learned through a denoising score network. Experiments on CIFAR-10 and ImageNet-100 show higher effective rank than SimCLR, but lower classification and transfer accuracy. The claimed computational advantages are mainly theoretical.
Critical strengths
- Interesting theoretical approach.
Replacing explicit negative-sample comparisons with a learned entropy-gradient estimate is an interesting idea. The theoretical connection is useful and could motivate further work on more efficient representation learning.
- Improved representation dimensionality.
ScoreCLR achieves a substantially higher effective rank than SimCLR (220 versus 96 on CIFAR-10). Although this does not improve downstream accuracy, it is an interesting result worth exploring further.
Critical weaknesses
- Limited practical benefits.
My main concern is that ScoreCLR consistently underperforms SimCLR. Linear-probe accuracy is lower on both CIFAR-10 (87.31% versus 89.45%) and ImageNet-100 (68.88% versus 72.60%), with similar results for transfer learning.
Despite its favourable theoretical complexity, ScoreCLR is also approximately 1.7 times slower per epoch in the reported experiments. The higher effective rank is interesting, but does not compensate for the lower accuracy and additional computational cost.
- Unclear connection between theory and implementation.
The theoretical derivation requires the marginal score, while the implementation uses a conditional score network, including conditioning on unrelated images during repulsion. It is unclear whether this provides the intended entropy-gradient estimate.
The reported training instabilities are another concern, particularly as results are based on a single random seed.
Questions for authors
Could the authors explain why the conditional score estimator approximates the marginal entropy gradient? Would an unconditional score network be more appropriate?
Could the authors demonstrate the claimed computational advantage at larger batch sizes, including actual training time and memory usage?
Could the authors report results across multiple random seeds, given the observed training instability?
Response to Reviewer 5dAD
We thank the reviewer for the careful review and the clear questions.
Main concern: limited practical benefits
Response. The revision reduces both the accuracy gap and the time overhead. Over three seeds, ScoreCLR reaches 88.87 ± 0.44 against 89.26 ± 0.38 for SimCLR on CIFAR-10 (was 87.31 against 89.45), a gap that three seeds do not separate, and 71.23 ± 0.06 against 72.39 ± 0.62 on ImageNet-100 (was 68.88 against 72.60). An epoch now takes 1.08× SimCLR's time on CIFAR-10 with one score-network step per iteration, against 1.7× in the submission with five steps. On ImageNet-100 the recipe uses five steps and an epoch takes now 1.01× SimCLR's time. ScoreCLR still does not outperform SimCLR, and the paper says so (Section 4.2): transfer is lower on every transfer target of the paper (STL-10, DTD and CIFAR-10), by 0.9 to 4.4 points (Table 4), and the higher effective rank has not translated into accuracy. On CIFAR-10, ScoreCLR is 0.8 points below LeJEPA, level with BYOL and 1.0 above Barlow Twins (Table 1; single runs for the last two). What we defend is an estimator, not a new state of the art: a repulsion without negatives that follows Prop. 3.1, close to SimCLR in accuracy and training time, with an objective whose cost is linear in the batch size.
Question 1
Response. It does not, and an unconditional network is the appropriate choice: a network conditioned on another latent returns a conditional score, not the marginal one. The score network is now unconditional, ε_θ(z_t, t) (Section 3.3). Putting the conditioning back lowers the linear accuracy from 89.38 / 88.58 to 83.83 / 85.27 (CIFAR-10, seeds 1/2; Appendix C.6; recipe not re-tuned for the conditional network): the network that Prop. 3.1 prescribes is also the more accurate one.
Question 2
Response. For the objective alone the advantage is large at large batch: at B=32768 it uses 44 times less memory than InfoNCE (1.1 GiB against 48.3 GiB) and, with S=10, it is 4.7 times faster. For a full ResNet-18 step the advantage is small, because the encoder dominates. Cost of the objective alone (one H100, fp32; Appendix C.7); S is the number of score-network updates per encoder step and K the number of noise queries per code; the large-batch runs need S=10:
Full training step (one H100, ScoreCLR / SimCLR):
A full ResNet-18 step therefore takes the same time as SimCLR and uses 2–11% less memory. We have not trained at B=32768: the last row measures cost only.
Question 3
Response. Table 1 now gives the mean ± std over seeds 1–3 (linear top-1): on CIFAR-10, 88.87 ± 0.44 (89.38 / 88.58 / 88.66) against 89.26 ± 0.38 for SimCLR; on ImageNet-100, 71.23 ± 0.06 (71.22 / 71.18 / 71.30) against 72.39 ± 0.62. No seed stalled or collapsed for the current method.