acceptodds
Withdrawn from ICLR 2027

A Contrastive Target Is a Demand, Not a Label

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

An InfoNCE target is usually read as a label: it says which samples belong together. We show that it acts as a demand, and that what an encoder learns depends on how the demand can be met, not only on what it says. We write the target of a batch as a matrix with two parameters, sharpness and specificity, and build targets that agree exactly on class correctness, sharpness and the class composition of every batch, with the temperature tuned for every arm. The cleanest test compares two targets that name a random class-mate as each row’s positive and carry identical information. When the class-mate is fixed, encoders memorise the pairing. When it is redrawn at every step, they cannot, and held-out classes become tighter by 30 to 50% in angular spread and more accurate by 1.5 to 4.6 points of withinview class retrieval. Against the standard target, which names the true partner, the redrawn class-mate gains 3.4 and 3.3 points of class structure on two image settings and gives up 44 and 82 points of instance retrieval, a trade that reappears at larger scale on CIFAR-100 with a ResNet-18. The second parameter, sharpness, behaves as a second temperature. Over a 7 × 8 grid with every optimum interior, the best temperature falls 40× as sharpness grows 73×, the best smoothed target is within 2 points of the best-tempered one-hot target in three of four settings, and a closed form of the loss predicts the direction of this ridge. With the temperature free, sharpness is harmless on a correct target, and most of the damage a sharp wrong target does in a fixed-temperature loss disappears. Specificity, the axis the temperature cannot reach, is where the target’s content lives.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
contrastive learning, InfoNCE, self-supervised learning, representation learning, soft targets, softmax temperature, false negatives, empirical analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qbz1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper contains an analysis of the InfoNCE loss function with a focus on exploring how varying (i) the way that positives and negatives are chosen and (ii) the sharpness of the positive vs. negative distribution impacts different aspects of downstream model performance. All experiments use either a dataset referred to as ‘digits’ or Fashion-MNIST. The paper is organized around explorations of (i) and (ii). To facilitate this, the paper introduces a parameter φ\varphi that controls how often the positive for a class is a variation of the same instance vs another instance in the class. The paper provides evidence that in the former case, the model learns features relevant to instance retrieval but in the latter case, representations more closely cluster in terms of class identity. The paper also introduces a way to vary the sharpness of the distribution through a parameter ρ∈(0,1]\rho \in (0,1]. The paper provides evidence that this sharpness plays a similar role to temperature and that sharp distributions can be detrimental if there is label noise for certain instances.

Critical strengths
  • Though the experiments are limited, the direction is interesting: The reviewer believes that the idea of introducing parameters that can disentangle different sources of variation in InfoNCE is a good approach to better understanding this approach to self-supervised learning. The specific parameters that this paper introduces are reasonable and get at the issues without introducing to much additional complexity.
Critical weaknesses
  • Clarity: The reviewer found this work very hard to read and understand. It is possible that some of this comes from the reviewer not being an expert in this area, but there are also basic practices that the paper could adopt to improve exposition and clarity. The value of ICLR papers does not come from the quality of their writing, but if the reader cannot understand the points the paper is making, that is a serious issue. As an example, the experimental set up should be very clearly stated. The purpose and motivation of various constructions should be made as clearly and simply as possible. Since the whole paper depends on Section 2, this should be expanded to ensure that the set-up is understood by the reader. It would be very helpful to also include a simple example. Another problem is the many points in the paper where vague statements are made and then never explained. For instance, on line 399 the paper says that “So “class structure was destroyed” can mean two things.” But what these two things are is never said. On line 307, the paper says “We reimplemented Eq. 1 from scratch instead of reusing our own trainer.” But then it never says how or why this was done. In general, the paper often reads as rough research notes. Finally, the paper often uses unnecessarily complex jargon like “granted-true-pairs diagnostic control”. There has to be a simpler way to say this (possible using more words).
  • Broader or deeper analysis: As noted above, the paper is structured around two specific experiments. While the paper does summarize the results and (vaguely) states conclusions, no further analysis is provided. There are a range of ways that this could be increased. For example, connecting the experimental conclusions to actual design considerations that are of importance to practitioners (e.g., how can we use these results to build better algorithms?) or analysis into why varying the chosen parameters results in the observed behavior. In particular, it may be that at least one of these experiments is amendable to theoretical analysis. As it is, the impact of this work seems like it will be very limited.
  • Impact: A result does not have to be surprising to be valuable. Confirming folk wisdom can be important. On the other hand, if a result largely confirms what is already expected, does not address an existing question, and does not deliver any additional insight, the value of the work can be questioned. To the reviewer, both of the results largely agree with community intuition. If you set the target to be another class element rather than a different view of the same instance, you will learn class features. Similarly, since sharpness of the distribution is closely related to temperature, it is not surprising that the effect of varying sharpness is very similar to the effect of varying temperature.
Questions for authors
  • Are there ways in which the results in this paper can inform best practices in self-supervised training or new methods that use InfoNCE?
  • What is the symbol after ρ/b\rho/b in (2)?
  • What is the ‘digits’ dataset, is this MNIST?
  • What does a ‘demand’ mean in this case?
  • What is the symbol before [j=π(i)][j = \pi(i)] in equation (2)?
Reviewer_bMJ1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is arguing that contrastive targets act as demands rather than semantic labels. In other words, they can influence the learning dynamics as optimization constraints geometrically. This introduces various confounders in designing contrastive losses. Focusing on InfoNCE, the paper claims the following:

  • Normal InfoNCE pairs positive and negative examples in pairs. This leads to memorization is the training dataset. Pairing examples with random examples of the same class (changing the specificity) removes memorization but also destroys quality.
  • De-sharpening the target matrix doesn't help much -- it basically acts as another temperature.
  • Temperature can act as an absorption filter towards noisy target matrices.

Essentially the paper is an empirical study on confounding factors introduced by varying the target InfoNCE matrix.

Critical strengths

The paper is a methodical study of the confounders that can be introduced by ablating the target matrix in contrastive learning. The study of such effects is important in informing the design and deployment of future contrastive learning techniques. The insights obtained are useful: semantic association is not necessarily going to improve learning -- it might instead reinforce memorization. Adjusting the temperature can help deal with noisy targets.

The experiments are performed soundly, and the questions examined are natural. The paper does a good job of visualizing the results and helping build intuition.

Critical weaknesses
  1. I found the text hard to read. Nouns are used as verbs a little too much ("the same target gains on class structure on both datasets"), many sentences are excessively long and dense. In general I think simplifying the prose would go a long way to improving the paper.

  2. I'm not very confident of the technical strength of the submission as an ICLR paper. It constitutes an important and relevant ablation study, but (a) I imagine the findings are more or less known to the community as effects of the learning dynamics under optimization constraints, (b) no new technique is introduced in the analysis of InfoNCE. There is little theoretical understanding of the reasons behind the observed phenomena.

Small nits:

  • Equation (2) has a typo.
Questions for authors

N/A

Reviewer_8effOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how contrastive targets shape learned representations by varying target sharpness, instance-versus-class pairing, and prediction temperature. Its controlled experiments show that fixed and resampled class-consistent pairings produce different memorization and representation behavior, and that the benefits of target smoothing depend on temperature tuning and auxiliary reconstruction losses.

Critical strengths

The paper provides useful controlled comparisons of target sharpness, pairing consistency, and temperature, while transparently reporting failed predictions and experimental limitations.

Critical weaknesses
  1. The incremental novelty is insufficiently established. SCE (§3.2, Eq. 6) already mixes one-hot and soft relational targets, while X-CLR (§3) formulates contrastive learning through target similarity graphs. ReSSL (Table 2) and SCE (Table 6) also jointly examine target and prediction temperatures. These use different target families and do not establish equivalence, but the paper should clarify what its controlled analysis establishes beyond these precedents.

  2. The experiments do not isolate memorization as the cause of the improvement. Sections 3–4 match per-step class correctness and batch composition, but resampling also changes the positive relationships encountered throughout training. The measured pairing memorization supports the proposed explanation, yet does not distinguish it from broader same-class pair coverage. A coverage-matched control or resampling-frequency study would help clarify this attribution.

  3. The central conclusions are insufficiently validated beyond small-scale settings. Appendix F uses 1,024 training samples, 384 validation samples, small MLPs, and one fixed data split; repeated seeds therefore do not measure variability across data draws. Although Appendix H extends the instance–class tradeoff to CIFAR-100, it does not repeat the fixed-versus-resampled comparison or the joint sharpness–temperature analysis. Whether these conclusions persist across sample sizes and model capacities remains unresolved; additional validation or narrower claims are needed.

Questions for authors
  1. What substantive insight does the proposed analysis add beyond SCE, ReSSL, and X-CLR, particularly regarding soft targets and joint temperature tuning?

  2. How can the gains from resampling be attributed to reduced memorization rather than increased same-class pair coverage? Can a coverage-matched control distinguish these explanations?

  3. Do the fixed-versus-resampled and sharpness–temperature findings persist with larger training sets, different model capacities, and independent data splits, beyond the partial CIFAR-100 validation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.