acceptodds
Under review as a conference paper at ICLR 2027

Residual Perturbations as Sentence-Pair Generators for Contrastive Sentence Embedding Training

Abstract

Sentence embeddings, which map sentences to dense vectors that capture their meaning, underpin many NLP tasks such as semantic search, clustering, and retrieval. Training sentence embeddings typically relies on sentence pairs, which are costly to generate. To solve this resource bottleneck, researchers have developed alternative methods to generate sentence pairs efficiently. Unsupervised alternatives bypass this by forming positive pairs from single sentences. SimCSE is a well-established unsupervised method that uses dropout noise to create two views of the same sentence as a positive pair. In this article, we introduce and explore residual perturbations as an alternative sentence pair generation method. Residual perturbations generate sentence pairs by adding perturbations onto the original embedding between the embedding layer and subsequent attention layers, treating the resulting perturbed embedding as the positive counterpart to the original. We investigate different embedding-level perturbations: random noise, rolling, shuffling, and adding the sentence's centroid. The results highlight that embedding-level residual perturbations performs on par with dropout-based methods, such as SimCSE. On the MTEB benchmark, residual rolling and residual shuffling perturbations on average match or slightly exceed SimCSE after unsupervised contrastive pretraining, with residual shuffling winning the largest share of individual tasks. Adding the centroid recovers much of this benefit through a simple constant shift, while random noise and, more severely, discarding the original embedding underperform even an untouched baseline. While our method is comparable to SimCSE overall, viewing performance per task category reveals clear differences in performance with narrow losses on retrieval and clustering tasks and strong improvements on STS and pair classification tasks. The strong performance of the residual shuffle perturbation on STS tasks suggests that perturbations can be chosen according to the downstream task at hand, offering an additional layer of control over sentence embedding generation, as well as a working alternative to dropout-based methods of sentence pair generation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
sentence embeddings, contrastive learning, unsupervised representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_XSCDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes residual perturbations applied at the BERT embedding layer as an alternative to dropout-based positive pair generation for unsupervised contrastive sentence embedding learning. The authors evaluate five perturbation variants on the MTEB benchmark, with SimCSE as the primary baseline. They report that residual rolling and shuffling achieve overall performance on par with SimCSE, improving on STS and pair classification tasks but declining on retrieval and clustering tasks.

Critical strengths

All perturbations are applied at the same point in the model, and the five variants are designed as pairwise controls: with vs. without structure, with vs. without the residual addition, and preserving vs. disrupting word order. This design supports the paper's main claims with the evidence presented.

Critical weaknesses
  1. Lack of academic rigor The introduction frames state-of-the-art sentence embedding research using only work from 2018–2019, while relevant recent advances are omitted. [1] is the related prior work: it also constructs perturbations for unsupervised contrastive sentence embedding learning, but its perturbations are learned rather than hand-crafted. It is neither cited nor used as a baseline, making it impossible to judge whether the proposed fixed residual perturbations offer any advantage. Similarly, [2] shows that sentence embeddings can be obtained with simple prompt-based strategies, which weakens the motivation for the proposed augmentation pipeline. More broadly, the related work section reads as a raw chronological list of papers with no synthesis or positioning of the proposed method against existing work. Without engaging with these closely related lines of research, the contribution of the paper remains unclear.

[1] https://openreview.net/pdf?id=gZYbGIpFYpA [2] https://arxiv.org/pdf/2201.04337

  1. Insufficient experimental validation The authors cite the finding that input-level cropping strongly outperforms SimCSE as motivation for this work, yet fail to include cropping as a baseline in their experiments. The proposed method is only compared to SimCSE (2021), with no evaluation against more recent state-of-the-art approaches. Without this key comparison, the results cannot demonstrate any meaningful advance over existing work, which undermines the paper’s novelty and practical value.
Questions for authors
  1. The authors cite the cropping-based augmentation method as motivation for their work, but it is not included as an experimental baseline. Why is this comparison missing?

  2. Key implementation details and hyperparameters are omitted from the paper. Please provide the full details required to replicate your experiments.

Reviewer_h5a7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies sentence embeddings. Existing sentence embeddings rely on costly sentence pairs for training, motivating alternatives like unsupervised methods that generate positive pairs from single sentences, e.g., SimCSE using dropout noise. This paper introduces residual perturbations, adding noise, rolling, shuffling, or centroid shifts at the embedding level, as a new sentence pair generation method, achieving performance on par with dropout-based approaches like SimCSE on the MTEB benchmark. Notably, residual shuffling matches or exceeds SimCSE overall, excels on STS and pair classification tasks, and allows task-specific perturbation selection, offering a flexible alternative to dropout-based methods.

Critical strengths
  • The paper is well written and easy to follow.
  • The perturbation suite is well designed and serves as a set of controlled comparisons. The five variants are not arbitrary: noise isolates the effect of "perturbing at all," residual mean isolates a "structure-free constant shift," and non-residual rolling isolates the contribution of the residual addition itself. This experimental logic makes the findings (the residual connection is essential; structure drawn from the sentence itself matters) interpretable.
  • The experiments extensively compare the proposed method against baselines and demonstrate its effectiveness, although the selected baselines and backbone are somewhat dated.
Critical weaknesses
  • The works discussed in the Introduction and Related Work sections are outdated. I would suggest that the authors conduct a more up-to-date literature review to reflect recent progress on sentence embeddings, especially how the field has evolved with the rise of LLMs.
  • The selected backbone is also too old. I would suggest that the authors experiment with small language models (e.g., Qwen2.5-0.5B, https://huggingface.co/Qwen/Qwen2.5-0.5B) to examine whether the proposed method also works for LLM architectures. Given the strong general capability of LLMs, simple training tricks such as noise adding or shuffling may no longer be very helpful; running these experiments on LLMs would therefore provide useful insights for frontier research on sentence embeddings.
  • According to Table 2, the SimCSE baseline still outperforms the proposed techniques on most task categories, all except two of the seven: clustering and STS. This suggests that the proposed techniques may not be particularly effective at improving overall sentence embedding quality.
Questions for authors
  • What are the batch size and InfoNCE temperature? Both are known to strongly affect contrastive learning, and the residual variants might have different sensitivity than dropout-based positives.
Reviewer_vyQnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a way to build positive pairs for unsupervised contrastive training of sentence embeddings. After BERT's embedding layer and before the first attention block, a perturbation (a rolled or shuffled copy of the token embeddings, the sentence centroid, or Gaussian noise) is added to the original embeddings, and the result is used as the positive view in place of SimCSE's dropout view. The encoder itself is unchanged. Models are trained on BookCorpus starting from pretrained BERT and evaluated on the 41 tasks of MTEB(eng, v2). Residual shuffling and rolling come out on par with SimCSE overall, with gains on STS and pair classification and losses on some other categories.

Critical strengths
  • The method is very simple. It doesn't touch the encoder and needs no extra modules or generators.
  • The evaluation uses the full MTEB(eng, v2) suite, and all per-task scores are reported, which makes the results easy to check.
  • The claims are stated carefully. The paper uses paired tests and bootstrap intervals and calls the overall result "on par" rather than overselling it.
  • The controls are sensible: rolling without the residual, residual mean as a constant shift, and an identical-view control.
Critical weaknesses
  1. Novelty and relevance are limited. Building positives by perturbing the embedding layer is already well studied: ConSERT used token shuffling and feature cutoff at the embedding layer in 2021. The new part here is mainly adding the perturbed copy back to the original, which is a small change. The paper also doesn't place itself relative to current sentence embedding methods. The only method baseline is SimCSE, the only backbone is BERT, and none of the other unsupervised methods it cites (ConSERT, ESimCSE, DiffCSE) are compared experimentally. Even if the results hold, it's hard to see what this means for sentence embeddings today.
  2. The SimCSE baseline looks under-performing, and this matters for the main positive result. The reproduced SimCSE is far below the numbers in the original paper on standard STS tasks (e.g. STS-B in the low 60s versus about 77). It even seems to get worse with more training data: in the smaller 1M-sample run (Table 6) SimCSE has a higher STS score than at full scale, and there residual shuffling is actually below SimCSE on STS, while residual rolling is only slightly ahead. So the significant STS gain in Table 5 may come largely from the training setup (BookCorpus, gradual unfreezing, and, as far as I can tell, no dev-based checkpoint selection) rather than from the perturbations. It would help to reproduce SimCSE in its original setting, train the residual variants the same way, and report the public SimCSE checkpoint on MTEB as a reference.
  3. The explanation of the mechanism doesn't fit how BERT works, and one result suggests a possible implementation issue. The perturbation is applied after the absolute position embeddings have already been added. After that point BERT is permutation-equivariant, and mean pooling is permutation-invariant. So plain rolling, as long as the attention mask is permuted consistently, should give exactly the same pooled representation as the original sentence (up to dropout), and should behave about like SimCSE. Instead it scores well below untrained BERT. This points to something like padding being rolled into the sentence or the mask not being permuted. It also means the "word-order invariance" explanation for the STS gains is not supported. The residual variants really mix two token/position vectors at each position, which is closer to an intra-sentence mixup, and no probing experiment checks the word-order story.
  4. It's unclear whether this is an alternative to dropout or an addition to it. The paper sets BERT dropout to 0.1 for all models and explicitly disables it only for the identical control. That suggests dropout is still on for the residual variants, so the two views differ by both dropout and the perturbation. The abstract and conclusion present the method as an alternative to dropout-based positives.
  5. The experimental scope is narrow. There is one backbone (whose size isn't stated), what appears to be a single training run per variant with no seed variance, and no ablation on the perturbation strength (e.g. a scale on the added copy, or the noise variance). The noise variant uses a single N(0,1) scale and ends up below BERT, so it isn't a very informative control.
  6. The statistics are weaker than they look. The tests treat the 41 MTEB tasks as independent paired samples, but many tasks are closely related (e.g. several STS sets from similar sources). The category-level tests in Table 5 cover three variants across several categories without any correction for multiple comparisons, and some of the significant results would likely not survive one. The task-level bootstrap also doesn't capture training randomness.
  7. Minor: several details needed for reproduction are missing, including the BERT size, batch size, temperature, optimizer and schedule details, whether the SwiGLU head is kept at evaluation, and how padding is handled when rolling or shuffling.
Questions for authors
  1. Are roll and shuffle applied to the padded sequence or only to the real tokens (and what about [CLS]/[SEP])? Are the attention mask and pooling mask permuted the same way? With a correct mask, does plain rolling give the same pooled output as the unperturbed input?
  2. Why is the reproduced SimCSE so much lower than in the original paper? Under the original SimCSE training setup, do the residual variants still improve STS?
  3. Is dropout enabled when training the residual variants, and do the two views get different dropout masks?
  4. Are the category-level differences (better on STS, worse on retrieval and clustering) stable across multiple seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.