acceptodds
Withdrawn from ICLR 2027

What Do Negatives Buy? A Controlled Anatomy of Contrastive vs. Negative-Free Text Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive negatives are the default for unsupervised sentence embeddings; what do they buy? We swap SimCSE's InfoNCE for a negative-free Gaussianity penalty with view prediction, holding encoder, data, views, pooling and projector fixed. This prices two tuned packages, not one term; we also intervene inside InfoNCE. On RoBERTa, negatives buy a trade-off rather than a uniform advantage: at development-selected checkpoints, they raise semantic similarity (STS) and cost topic clustering. The cost appears at every corpus size in our main study; without selection it attenuates at our largest corpus. At our smallest corpus its sign holds in every architecture clearing our competence bar, decisively in all but the smallest. Under RoBERTa's main recipe the STS benefit grows with training scale, which one pass ties to corpus size. Fresh pre-registered seeds replicate this growth in sign, with no learning-rate dependence demonstrated; a pre-registered rerun at a lower negative-free learning rate did not demonstrate the growth. BERT, at optima from a one-seed search, already shows the benefit at our smallest corpus. On RoBERTa at matched count, dropping same-topic rather than random negatives raises clustering, even with labels from no neural model. Our pre-registered headline, that InfoNCE degrades at small batch, was not supported.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
sentence embeddings, contrastive learning, InfoNCE, SimCSE, negative-free self-supervised learning, negative sampling, controlled comparison, representation geometry, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1B38OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares SimCSE against a negative-free SIGReg objective in a shared pipeline. Within InfoNCE, it varies how many negatives are kept and whether negatives in same topics are removed. On RoBERTa-base, it finds that negatives raise STS but hurt clustering at development-selected checkpoints, and keeping more negatives reproduces this trade. Dropping same topic rather than random negatives improves clustering. Uniformity and a distributional shape statistic forecast the two sides within a backbone.

Critical strengths

None.

Critical weaknesses
  1. The paper is extremely hard to read, and the main argument depends on material outside the submission. The main text reads as a summary of a lab notebook. Nearly every sentence carries a status mark (⋆, *, ○, †, ↓, “desc.”), or a cross-reference.
  2. The package comparison is confounded by tuning and checkpoint selection. The two objectives received different tuning budgets: SimCSE got a 6-trial lr × τ grid at 1M, while SIGReg got 8 λ values at 250k, and SIGReg’s learning rate was never swept.
Questions for authors

Could the essential evidence be moved into the paper’s own appendix instead of rest only on the Extended Materials?

Reviewer_VW6DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper experiments with using view prediction + SIGReg's Gaussianity penalty instead of InfoNCE term in SimCSE to study the role of negatives in contrastive learning. The authors run three experiments under this experimental framework and report that negatives (as opposed to their approach) lead to lower clustering results (but higher STS results).

Critical strengths

None

Critical weaknesses

The results are not interesting or novel, by the author's own admission this has been studied (see lines 035-037) and similar results have been found, the authors make no attempt to differentiate their results. The experiments are extremely limited, focused on one architecture with one dataset when constrastive learning is extremely popular and many alternatives were available. The presentation is extremely poor and difficult to follow.

Questions for authors

None

Reviewer_82QKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the role of negatives in unsupervised text embedding learning. The authors compare SimCSE with a negative-free objective based on SIGReg under a largely controlled training pipeline, and further examine the effects of negative count and composition within InfoNCE. The main findings suggest that contrastive negatives improve STS tasks but can hurt clustering performance, while removing same-topic negatives can improve clustering compared to removing an equal number of random negatives.

Critical strengths
  1. The paper investigates an interesting question about the role of negatives in contrastive representation learning. The controlled comparisons, including the interventions on negative count and composition within InfoNCE, provide useful empirical observations.
  2. The paper conducts extensive experiments across different configurations, including multiple random seeds, model architectures, and checkpoint selection strategies. The authors also report negative and inconclusive results, such as the unsupported small-batch hypothesis and the failure of some findings to generalize across model architectures.
Critical weaknesses
  1. The trade-off between semantic similarity and clustering, as well as the potential harm of false negatives, has been discussed in prior contrastive learning literature. Many observations are therefore relatively expected, and the paper does not provide substantially new insights into the underlying mechanisms. For example, the improvement from removing same-topic negatives may largely reflect their contribution to the softmax mass rather than their semantic identity.
  2. The main findings rely heavily on specific STS tasks and clustering tasks, whose evaluation objectives do not necessarily reflect general embedding quality. Importantly, clustering performance can be highly sensitive to the definition and granularity of evaluation labels. Similarly, the effects of negatives likely depend on the semantic diversity and distribution of the training data, as well as batch composition, but these factors are not sufficiently disentangled. Consequently, it remains unclear how broadly the reported trade-offs hold beyond the specific experimental settings. It would be useful to evaluate clustering tasks with different label granularities and further investigate the effects of training data distribution, e.g., by measuring semantic diversity and same-topic collision rates across different training corpora.
Questions for authors

see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.