acceptodds
Under review as a conference paper at ICLR 2027

Contrastive latent representation learning with denoising score matching

Abstract

Contrastive representation learning methods operate by pulling different views of the same sample together while pushing them away from views of different samples. A fundamental challenge is the estimation of the repulsive force, which typically requires large batch sizes, can exhibit high variance, and is susceptible to the curse of dimensionality. We introduce ScoreCLR, a contrastive representation learning method that offers a novel way to estimate this repulsion term. We consider the framework of representation learning by maximizing the mutual information between two latent views of a sample. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we obtain an equivalent objective that involves the score of the latent representation. We then use denoising score matching techniques to estimate this score with a lightweight network that is trained jointly with the encoder. We further propose regularization techniques using different noise levels and a Gaussian prior on the latent representation. ScoreCLR replaces the batch of explicit negatives of contrastive methods with a learned, amortized repulsion whose cost is linear in the batch size rather than quadratic. On CIFAR-10 and ImageNet-100 it reaches linear-probe accuracy within 2–4 points of SimCLR under a shared recipe, while producing projector outputs of higher effective rank.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, contrastive learning, representation learning, mutual information, score matching, denoising score matching, diffusion models, curse of dimensionality, latent representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5dADOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ScoreCLR, a self-supervised learning method that replaces contrastive repulsion with an estimated gradient of latent entropy, learned through a denoising score network. Experiments on CIFAR-10 and ImageNet-100 show higher effective rank than SimCLR, but lower classification and transfer accuracy. The claimed computational advantages are mainly theoretical.

Critical strengths
  1. Interesting theoretical approach.

Replacing explicit negative-sample comparisons with a learned entropy-gradient estimate is an interesting idea. The theoretical connection is useful and could motivate further work on more efficient representation learning.

  1. Improved representation dimensionality.

ScoreCLR achieves a substantially higher effective rank than SimCLR (220 versus 96 on CIFAR-10). Although this does not improve downstream accuracy, it is an interesting result worth exploring further.

Critical weaknesses
  1. Limited practical benefits.

My main concern is that ScoreCLR consistently underperforms SimCLR. Linear-probe accuracy is lower on both CIFAR-10 (87.31% versus 89.45%) and ImageNet-100 (68.88% versus 72.60%), with similar results for transfer learning.

Despite its favourable theoretical complexity, ScoreCLR is also approximately 1.7 times slower per epoch in the reported experiments. The higher effective rank is interesting, but does not compensate for the lower accuracy and additional computational cost.

  1. Unclear connection between theory and implementation.

The theoretical derivation requires the marginal score, while the implementation uses a conditional score network, including conditioning on unrelated images during repulsion. It is unclear whether this provides the intended entropy-gradient estimate.

The reported training instabilities are another concern, particularly as results are based on a single random seed.

Questions for authors

Could the authors explain why the conditional score estimator approximates the marginal entropy gradient? Would an unconditional score network be more appropriate?

Could the authors demonstrate the claimed computational advantage at larger batch sizes, including actual training time and memory usage?

Could the authors report results across multiple random seeds, given the observed training instability?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5dAD

We thank the reviewer for the careful review and the clear questions.

Main concern: limited practical benefits

Response. The revision reduces both the accuracy gap and the time overhead. Over three seeds, ScoreCLR reaches 88.87 ± 0.44 against 89.26 ± 0.38 for SimCLR on CIFAR-10 (was 87.31 against 89.45), a gap that three seeds do not separate, and 71.23 ± 0.06 against 72.39 ± 0.62 on ImageNet-100 (was 68.88 against 72.60). An epoch now takes 1.08× SimCLR's time on CIFAR-10 with one score-network step per iteration, against 1.7× in the submission with five steps. On ImageNet-100 the recipe uses five steps and an epoch takes now 1.01× SimCLR's time. ScoreCLR still does not outperform SimCLR, and the paper says so (Section 4.2): transfer is lower on every transfer target of the paper (STL-10, DTD and CIFAR-10), by 0.9 to 4.4 points (Table 4), and the higher effective rank has not translated into accuracy. On CIFAR-10, ScoreCLR is 0.8 points below LeJEPA, level with BYOL and 1.0 above Barlow Twins (Table 1; single runs for the last two). What we defend is an estimator, not a new state of the art: a repulsion without negatives that follows Prop. 3.1, close to SimCLR in accuracy and training time, with an objective whose cost is linear in the batch size.

Question 1

Could the authors explain why the conditional score estimator approximates the marginal entropy gradient? Would an unconditional score network be more appropriate?

Response. It does not, and an unconditional network is the appropriate choice: a network conditioned on another latent returns a conditional score, not the marginal one. The score network is now unconditional, ε_θ(z_t, t) (Section 3.3). Putting the conditioning back lowers the linear accuracy from 89.38 / 88.58 to 83.83 / 85.27 (CIFAR-10, seeds 1/2; Appendix C.6; recipe not re-tuned for the conditional network): the network that Prop. 3.1 prescribes is also the more accurate one.

Question 2

Could the authors demonstrate the claimed computational advantage at larger batch sizes, including actual training time and memory usage?

Response. For the objective alone the advantage is large at large batch: at B=32768 it uses 44 times less memory than InfoNCE (1.1 GiB against 48.3 GiB) and, with S=10, it is 4.7 times faster. For a full ResNet-18 step the advantage is small, because the encoder dominates. Cost of the objective alone (one H100, fp32; Appendix C.7); S is the number of score-network updates per encoder step and K the number of noise queries per code; the large-batch runs need S=10:

BInfoNCEScoreCLR S=1, K=1S=5, K=3S=10, K=1Memory: InfoNCE / ScoreCLR
1280.62 ms2.3 ms8.2 ms15.1 ms2 MiB / 5 MiB
20481.4 ms2.5 ms8.6 ms14.9 ms208 MiB / 71 MiB
40964.4 ms2.5 ms8.9 ms15.2 ms800 MiB / 141 MiB
819214.7 ms2.4 ms9.1 ms14.8 ms3.1 GiB / 281 MiB
32768228 ms7.6 ms30.3 ms48.5 ms48.3 GiB / 1.1 GiB

Full training step (one H100, ScoreCLR / SimCLR):

SettingTimePeak memory
B=4096, one epoch, fp16, S=108.27 s / 8.28 s21.7 / 22.2 GiB (2% less)
B=8192, one epoch, fp16, S=1010.00 s / 9.98 s43.1 / 45.8 GiB (6% less)
B=32768 over 8 GPUs, step of one GPU (emulated), fp32, S=5508 ms / 531 ms43.1 / 48.7 GiB (11% less)

A full ResNet-18 step therefore takes the same time as SimCLR and uses 2–11% less memory. We have not trained at B=32768: the last row measures cost only.

Question 3

Could the authors report results across multiple random seeds, given the observed training instability?

Response. Table 1 now gives the mean ± std over seeds 1–3 (linear top-1): on CIFAR-10, 88.87 ± 0.44 (89.38 / 88.58 / 88.66) against 89.26 ± 0.38 for SimCLR; on ImageNet-100, 71.23 ± 0.06 (71.22 / 71.18 / 71.30) against 72.39 ± 0.62. No seed stalled or collapsed for the current method.

Reviewer_avctOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ScoreCLR, which replaces explicit negative samples with a learned latent score. An entropy-gradient identity motivates the repulsion term, estimated through denoising score matching. Experiments examine classification, transfer, representation geometry and training cost.

Critical strengths
  1. The entropy-gradient formulation offers an interesting way to learn repulsion without a pairwise similarity matrix.
  2. The paper clearly separates backbone and projector geometry, rather than treating higher rank as better accuracy.
  3. The reporting is candid about accuracy gaps, training instability and practical runtime.
Critical weaknesses
  1. Proposition 3.1 uses the marginal score, but the denoiser learns with same-image conditioning and is queried with another image.
  2. The motivation emphasizes batch-size dependence and high-dimensional estimation, but experiments use only 256-dimensional outputs and batch sizes of 128 or 256.
  3. The memory analysis assumes 16-bit activations and five score-network steps, while ScoreCLR requires fp32 and the main ImageNet-100 recipe uses ten steps.
Questions for authors
  1. How does mismatched conditioning preserve the entropy-gradient interpretation?
  2. When does ScoreCLR outperform alternatives at matched training cost?
  3. Are the main results stable across seeds?
  4. ImageNet-100 linear accuracy is 68.88% versus 72.60% for matched-recipe SimCLR; DTD transfer is 53.72% versus 60.16%. In what setting does the learned repulsion offer a useful advantage?
2 comments
AuthorsOpenReview ↗

Response to Reviewer avct

We thank the reviewer for the careful review and the precise questions.

Question 1

How does mismatched conditioning preserve the entropy-gradient interpretation?

Response. It does not. A network conditioned on another latent returns a conditional score, not the marginal score that Prop. 3.1 requires. The score network is now unconditional (Section 3.3), so the implemented repulsion is the estimator of Prop. 3.1. for the noised latent. Putting the conditioning back loses 3.3–5.6 points of linear accuracy (83.83 / 85.27 against 89.38 / 88.58, CIFAR-10, seeds 1/2; Appendix C.6), without re-tuning the recipe for the conditional network.

Weaknesses 2 and 3

Response. Batch size: both methods now run from B=128 to 4096 (Section 4.5). ScoreCLR is within 0.9 points of SimCLR up to B=2048 and 1.4 below at 4096. Dimension: all runs use 256-dimensional codes, so the experiments do not test the high-dimensional argument, which Section 2.1 presents only as a possibility. Precision and steps: ScoreCLR now trains in fp16 without loss (88.82 ± 0.04 against 88.87 ± 0.44 in fp32 on CIFAR-10, Section 4.6), and the ImageNet-100 recipe uses five score-network steps instead of ten. Memory is now measured (Section 4.7, Appendix C.7): the objective uses 44 times less memory than InfoNCE at B=32768 (1.1 GiB against 48.3 GiB); in a full ResNet-18 training step, where the encoder dominates, this gives 6% less memory at B=8192 in fp16 and 11% at B=32768 in fp32.

Question 2

When does ScoreCLR outperform alternatives at matched training cost?

Response. Against SimCLR it does not, in the setting we ran. The cost is matched: an epoch takes 1.08× SimCLR's time on CIFAR-10 and 1.01× on ImageNet-100. At that cost ScoreCLR is 0.4 points below SimCLR on CIFAR-10, less than the variation between seeds, and 1.2 below on ImageNet-100. On CIFAR-10 it is 0.8 below LeJEPA, level with BYOL and 1.0 above Barlow Twins (Table 1; single runs for the last two). What differs is the geometry of the projector output z: its effective rank is 229 of 256 on CIFAR-10, against 96 for SimCLR and 58 for LeJEPA, and it is closer to a Gaussian (estimated Poincaré constant 7.8, against 25.0 and 13.5, and 1 for a Gaussian; Section 4.3). This does not translate into probe or transfer accuracy, and we have not yet tested a task that uses z itself.

Question 3

Are the main results stable across seeds?

Response. Yes. Table 1 reports the mean ± std over seeds 1–3 of the linear top-1 accuracy (k-NN in parentheses):

SimCLRScoreCLR
CIFAR-1089.26 ± 0.38 (88.73 ± 0.18)88.87 ± 0.44 (88.27 ± 0.13)
ImageNet-10072.39 ± 0.62 (66.41 ± 0.44)71.23 ± 0.06 (66.05 ± 0.39)

No seed stalled or collapsed. Both recipes use the gradient clip of the baselines (1.0); with it, ImageNet-100 is stable even with three queries (71.22 / 71.18 / 71.30).

Question 4

ImageNet-100 linear accuracy is 68.88% versus 72.60% for matched-recipe SimCLR; DTD transfer is 53.72% versus 60.16%. In what setting does the learned repulsion offer a useful advantage?

Response. With the new recipe the ImageNet-100 gap is 1.2 points (71.23 ± 0.06 against 72.39 ± 0.62) and the DTD gap 3.1 points (56.3 against 59.4); transfer stays below SimCLR on every target, by up to 4.4 points (Table 4). The learned repulsion offers two things.

First, memory at large batch: the objective uses 281 MiB against 3.1 GiB for InfoNCE at B=8192, and 1.1 GiB against 48.3 GiB at B=32768 (Section 4.7). With the ten score-network updates needed at large batch, it is still faster beyond B≈8k. In a full ResNet-18 step the encoder dominates, so this large saving of the objective becomes 2–11% of the step's memory.

Second, the geometry of z: on ImageNet-100 its effective rank is 244 of 256 against 143 for SimCLR (Tables 3). On CIFAR-10, its relatively small estimated Poincaré constant (Table 3) promises easier generative modeling of the latent, but we have not tested it.

Reviewer_avctOpenReview ↗

Response

Thank you for your response. A clear practical benefit over existing methods remains to be demonstrated, and the potential advantage in higher-dimensional representations is still untested. So I maintain my rating.

Reviewer_oXeFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ScoreCLR, a method for self-supervised representation learning. The approach estimates the repulsion term of the InfoNCE loss with a network trained by denoising score matching on the embeddings, together with a Gaussian prior that regularizes the embeddings. Since the repulsion term no longer requires pairwise comparisons, the cost of the objective is linear rather than quadratic in batch size. Results on CIFAR-10 and ImageNet-100 with a ResNet-18 show that ScoreCLR achieves performance within 2–4 points of the InfoNCE loss.

Critical strengths
  1. Interesting idea: using denoising score matching on the embeddings to estimate the repulsion term is a novel way to connect diffusion-based score estimation with contrastive learning.
  2. Memory-efficient loss function: since there are no pairwise comparisons, the loss scales as O(n) instead of O(n²) in batch size. However, this advantage is only analytical and only materializes at very large batches (e.g. CLIP like training which this paper didn't test)
Critical weaknesses
  1. The method still uses negatives indirectly: when computing s̄ (Algorithm 1, line 4), the score network is conditioned on the latent of another image, so each sample is effectively repelled from another sample’s latent. In addition, there is no evidence for the claim that the approach reduces the need for large batches as in InfoNCE: batch size is fixed (128 / 2×128) and never ablated.

  2. The method does not outperform InfoNCE: linear-probe accuracy is 2.1–3.7 points below SimCLR under the shared recipe and 3.5–6.5 points below SimCLR with LARS (Table 1), and it is worse on all out-of-domain transfer targets (Table 5). Evaluation is limited to CIFAR-10 and ImageNet-100 with ResNet-18, all from single-seed runs.

Questions for authors

Suggestions:

  1. Would it be possible to formulate the method without conditioning the score network on c (Algorithm 1, line 4)? Since c is the latent of another image at query time, this conditioning effectively reintroduces a negative pair, and Prop. 3.1 assumes the unconditional score of the latent.

  2. Could you add ablations over batch size, including larger batches (e.g., B ≥ 4096, where the objective is claimed to become cheaper)? Reporting the measured memory and wall-clock time alongside accuracy would directly support the scalability claim.

Questions:

  1. Do you think the embeddings learned with your approach could be useful for non-discriminative tasks such as image generation or tokenization? Given the higher effective rank and closer-to-Gaussian projector output, how would you expect them to perform in that setting compared to SimCLR?

  2. How does your method compare to other self-supervised methods such as BYOL and Barlow Twins, in terms of both downstream accuracy and memory consumption?

2 comments
AuthorsOpenReview ↗

Response to Reviewer oXeF (1/2)

We thank the reviewer for the careful review and the constructive suggestions.

Suggestion 1

Would it be possible to formulate the method without conditioning the score network on c (Algorithm 1, line 4)? [...] Prop. 3.1 assumes the unconditional score of the latent.

Response. We have followed this suggestion. The conditioning was a heuristic to ease the training of the score network, and the reviewer is right that it reintroduced a negative pair at query time. The new score network is unconditional, ε_θ(z_t, t), trained on the detached codes of the batch; no other sample enters the query (Section 3.3, Appendix C.2). Putting the conditioning back in the new recipe (CIFAR-10, seeds 1/2) gives 83.83 / 85.27 linear against 89.38 / 88.58 (Appendix C.6). The recipe was not re-tuned for the conditional network.

Suggestion 2

Could you add ablations over batch size, including larger batches (e.g., B ≥ 4096 [...])? Reporting the measured memory and wall-clock time alongside accuracy would directly support the scalability claim.

Response. We have added both. Linear accuracy on CIFAR-10 (Table 5; one run per cell, seed 5); S is the number of score-network updates per encoder step:

B12851220484096
ScoreCLR (S)88.80 (1)88.51 (5)88.29 (10)87.50 (10)
SimCLR89.4589.3288.6388.88

ScoreCLR is within 0.9 points of SimCLR up to B=2048, and 1.4 below at 4096. Large batches leave few score-network updates: at B=2048, S=1 is not sufficient, S=5 gives 87.84 and S=10 gives 88.29; at B=4096, S=5 gives 85.76 and S=10 gives 87.50.

Cost of the objective alone (one H100, fp32, time per step; Appendix C.7). K is the number of noise queries per code; the extra memory of ScoreCLR depends on neither S nor K:

BInfoNCEScoreCLR S=1, K=1S=5, K=3S=10, K=1Memory: InfoNCE / ScoreCLR
1280.62 ms2.3 ms8.2 ms15.1 ms2 MiB / 5 MiB
20481.4 ms2.5 ms8.6 ms14.9 ms208 MiB / 71 MiB
40964.4 ms2.5 ms8.9 ms15.2 ms800 MiB / 141 MiB
819214.7 ms2.4 ms9.1 ms14.8 ms3.1 GiB / 281 MiB
32768228 ms7.6 ms30.3 ms48.5 ms48.3 GiB / 1.1 GiB

The memory of the objective is lower from B≈1k and 44 times lower at B=32768. Its time grows slowly with B: it is faster than InfoNCE from B≈4k with S=1 and from B≈8k with S=5 or S=10, and 5 to 30 times faster at B=32768. In a full ResNet-18 step the encoder dominates, so this large saving of the objective becomes 2–11% of the peak memory. Measured on one H100, ScoreCLR / SimCLR:

SettingTimePeak memory
B=4096, one epoch, fp16, S=108.27 s / 8.28 s21.7 / 22.2 GiB (2% less)
B=8192, one epoch, fp16, S=1010.00 s / 9.98 s43.1 / 45.8 GiB (6% less)
B=32768 over 8 devices, one step of one device (emulated on one GPU), fp32, S=5508 ms / 531 ms43.1 / 48.7 GiB (11% less)

We have not trained at B=32768: the last row measures cost only.

AuthorsOpenReview ↗

Response to Reviewer oXeF (2/2)

Weaknesses: accuracy, single seed, LARS, transfer, scope, large-batch claim

Response. ScoreCLR still does not outperform InfoNCE. Over three seeds, the gap to SimCLR under the shared recipe is now 0.4 points on CIFAR-10 (88.87 ± 0.44 against 89.26 ± 0.38) and 1.2 on ImageNet-100 (71.23 ± 0.06 against 72.39 ± 0.62); it was 2.1 and 3.7 on one seed. Against SimCLR with LARS the gap is 1.9 and 4.2 points (it was 3.5 and 6.5); these rows use solo-learn's own pipeline, in which we did not train ScoreCLR, so they are reference points rather than a controlled comparison. ScoreCLR does not reduce the batch size needed for a given accuracy, and the paper now says so (Section 4.5). What remains is a repulsion without negatives that follows Prop. 3.1, within 0.4 to 1.2 points of SimCLR at almost the same epoch time, with a projector output of effective rank 229–244 of 256 against 96–143 for SimCLR.

Question 1

Do you think the embeddings learned with your approach could be useful for non-discriminative tasks such as image generation or tokenization? Given the higher effective rank and closer-to-Gaussian projector output, how would you expect them to perform in that setting compared to SimCLR?

Response. We have not run such an experiment, so this is our expectation rather than a result. The Poincaré constant equals 1 for a standard Gaussian and measures how easily a score-based model of the distribution can be learned and sampled (Koehler et al., 2023). Section 4.3 estimates it (Pillaud-Vivien et al., 2020): 7.8 for ScoreCLR against 25.0 for SimCLR and 13.5 for LeJEPA on CIFAR-10, an indication, not a proof, that our latent is closer to a Gaussian. We therefore expect a score or flow model on our latent to be easier to learn and to sample. Moreover, since our framework learns the score of the latent at several noise levels, it would be interesting to see whether this score network can be used for generation.

Question 2

How does your method compare to other self-supervised methods such as BYOL and Barlow Twins, in terms of both downstream accuracy and memory consumption?

Response. We have run both in our shared harness (CIFAR-10, seed 1; Table 1): BYOL reaches 88.90 linear accuracy and Barlow Twins 87.89, against 88.87 ± 0.44 for ScoreCLR over three seeds (89.38 on seed 1) and 89.26 ± 0.38 for SimCLR. With one seed we do not rank ScoreCLR against BYOL. Our harness removes BatchNorm from their heads, which penalizes them: BYOL reaches 89.96 with a BatchNorm predictor (Appendix C.1). Memory: the objectives of BYOL and Barlow Twins are already linear in B, so ScoreCLR has no memory advantage over them; its advantage is over pairwise objectives such as InfoNCE. It adds a small network of 0.28M parameters (2.2% of the encoder and projector).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers. The reviews share three criticisms: (i) the score network was conditional, so it did not estimate the marginal score of Prop. 3.1; (ii) every result was a single seed; (iii) the O(B) cost was only analytical. The revision addresses (i) to (iii). Its main change answers the concern raised in all three reviews: the score network is now unconditional, as Prop. 3.1 prescribes, and it is also the more accurate choice (Appendix C.6). Prop. 3.1, the form of the loss, the alternating training loop, the datasets and the backbone are those of the submission. The other changes answer requests made in the reviews (seeds, measured cost, batch-size study, BYOL and Barlow Twins).

Submitted (seed 5)Revised (seeds 1–3)
CIFAR-10 linear, ScoreCLR / SimCLR87.31 / 89.4588.87 ± 0.44 / 89.26 ± 0.38
ImageNet-100 linear, ScoreCLR / SimCLR68.88 / 72.6071.23 ± 0.06 / 72.39 ± 0.62
Epoch time relative to SimCLR1.7× (CIFAR-10)1.08× (CIFAR-10), 1.01× (ImageNet-100)
Precisionfp32 requiredfp16 equal to fp32 (CIFAR-10)

The high effective rank of the projector output is conserved (Section 4.3).

What we claim. (1) A repulsion without negatives that is, as implemented, the entropy-gradient estimator of Prop. 3.1 (Section 3.3) for the noised latent. (2) Accuracy close to SimCLR at the same training cost: 0.4 points below on CIFAR-10, which three seeds do not separate, and 1.2 below on ImageNet-100 (Table 1). (3) An objective whose cost is linear in the batch size B: at B=32768 it uses 1.1 GiB against 48.3 GiB for InfoNCE, 44 times less memory (Section 4.7). (4) A projector output that uses almost all of its dimensions: effective rank 229–244 of 256, against 96–143 for SimCLR and 58 for LeJEPA (Section 4.3).

What we do not claim. Better accuracy or transfer than SimCLR: transfer is 0.9 to 4.4 points lower (Table 4). A large saving for a full ResNet-18 step: the objective itself is 44 times lighter in memory at B=32768, but the encoder dominates a training step, so at B ≥ 4096 a full step takes the same time and 2–11% less memory. A demonstrated use of the geometry of z.

Other changes (Appendix C). ScoreCLR recipe, re-tuned: one noise query per code instead of three on CIFAR-10; score-network steps per iteration 5 → 1 on CIFAR-10 and 10 → 5 on ImageNet-100; invariance weight 10 → 2 and noise width 0.8 → 1.0 on ImageNet-100; a score network of 0.28M parameters (2.2% of the encoder and projector) instead of 0.38M (3.0%), which predicts the noise with an unweighted loss. Protocol: SimCLR, LeJEPA and ScoreCLR were re-run with seeds 1–3 in one code base with one augmentation pipeline. BYOL, Barlow Twins, SimCLR with LARS and the batch-size study are single runs. The ablations and the normality table of the submission are replaced by the ablations of the new recipe (Appendix C.6) and by an estimate of the Poincaré constant (Section 4.3). The cost figure is now measured; the analytical table stays in Appendix C.7. Section and table numbers in our replies refer to the revised PDF. An copy of the code, with the configurations of every run in the paper, is provided in the supplementary material.

Correction in Section 2.1. The submission stated that plugging the cosine critic fτf_\tau into the Donsker–Varadhan bound recovers the InfoNCE loss, and that symmetrizing it recovers the SimCLR loss. Both statements were incomplete. The revised Section 2.1 uses the critic f(z1,z2)=fτ(z1,z2)−log⁡E[efτ(z1,Z~2)]f(z_1,z_2)=f_\tau(z_1,z_2)-\log\mathbb{E}\big[e^{f_\tau(z_1,\tilde Z_2)}\big], for which the logarithm term of the bound vanishes. The resulting loss is closely related to InfoNCE, without the positive pair in the denominator, as in decoupled contrastive learning (Yeh et al., 2022). SimCLR is a close variant that symmetrizes over the two views, uses both views of the other samples as negatives and keeps the positive pair in the denominator. The former Sections 2.1 and 2.2 are merged, so Neural methods is now Section 2.2. The method and the results are unaffected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.