acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Pretraining for Data-Efficient Text Embeddings

Abstract

We compare bidirectional masked language modeling (MLM), bidirectional absorbing-mask discrete diffusion, and causal next-token prediction using parameter-matched ModernBERT-style models at four scales from 20M to 500M parameters, each pretrained on the same Ettin data mixture with a nominal budget of approximately 20 input tokens per parameter. We evaluate unsupervised SimCSE, frozen-backbone adaptation, and supervised InfoNCE fine-tuning on 10k, 100k, or 1.06M AllNLI and MS MARCO triplets, alongside raw-representation and random-initialization controls. We evaluate on seven STS benchmarks, four BEIR datasets, 20-Newsgroups clustering, typo robustness, and all 41 tasks of MTEB English v2. Under full supervision, the objectives reach similar MTEB means; the causal model leads local STS and clustering at every tested scale, and local retrieval from 60M onward. The objectives nevertheless separate in two ways. First, the diffusion–MLM retrieval gap narrows at larger scales and reverses at 500M on BEIR retrieval. Second, at 60M–500M, both bidirectional models outperform the causal baseline on local retrieval under SimCSE or 10k-triplet fine-tuning. Diffusion leads at 500M, with BEIR nDCG@10 of 0.229 versus 0.217 for MLM and 0.128 for the causal model under SimCSE, and 0.250 versus 0.243 and 0.135, respectively, with 10k triplets. With frozen-backbone adaptation, diffusion leads local STS and retrieval at 60M and 150M, whereas the causal model leads at 500M. Our results suggest that objectives that tie on aggregate embedding quality can differ systematically in retrieval, and that the direction of the difference depends on scale and on how much supervision the adaptation stage provides.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
NLP, Embeddings, Diffusion, Transformer, Representation, Metric Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7aNsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the question of whether diffusion pretraining improves downstream performance for language models adapted for information retrieval via contrastive fine tuning. The authors perform a broad experimental analysis of the question, considering matched parameter size, chinchilla optimally trained models trained with different pretraining objectives (MLM, causal LM, diffusion) as base models for subsequent IR fine tuning. The authors find that the diffusion objective is advantageous for larger-scale language models that are adapted for IR in a data limited regime, but that the advantage deteriorates with fine tuning dataset size.

Critical strengths
  • The problem of pretraining recipe for downstream IR fine tuning is interesting and relevant to the field.
  • The breadth of the paper's empirical experiments, from pretrianing configurations, to fine-tuning configuration, to evaluation breadth, is unusually strong.
Critical weaknesses
  • The abstract itself does not frame a problem, but rather, presents a laundry list of findings. The paper would benefit from a clear problem framing within which the findings are situated. The introduction itself does this, so much of the framing regarding pretraining for IR models can likely be applied in the abstract.
  • The largest weakness of the paper is that all experiments use a consistent pretraining corpus, which makes it difficult to generalize the findings to models trained with other pretraining corpora. The paper could be made much stronger if different pretraining corpora were considered.
Questions for authors

N/A

1 comment

Thank you for suggesting a second corpus and clearer problem framing.

Weakness 2. We additionally trained all three 150M models from scratch on 3B FineWeb-Edu tokens. We kept the tokenizer, model settings, token budget, optimizer settings, and adaptation procedure the same. Table 14(b) (provided in the updated manuscript, page 18, lower half) shows that diffusion also performs better than MLM on this second corpus: 0.25110.2511 versus 0.23520.2352 with 100k triplets, and 0.25900.2590 versus 0.24410.2441 with full supervision. Appendix B (provided in the updated manuscript, page 13, lower part) describes the data preparation. The added paragraph in Results summarizes these findings.

Weakness 1. We revised the opening of the abstract to state the practical problem: choosing a pretraining objective when labeled data for adaptation are limited. We also shortened the list of experimental details so that the research question is easier to follow.

Reviewer_NDytOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how three pretraining approaches, masked language modeling, diffusion, and next-token prediction affect the quality of text embeddings. The authors train models from 20M to 500M parameters using the same data and token budgets, then evaluate them under different amounts of downstream supervision. The results suggest that diffusion can provide a better starting point for retrieval when labeled data are scarce, particularly at the largest model size. With more supervised training, however, the causal models catch up or outperform diffusion on several tasks, while all three approaches achieve similar overall benchmark scores.

Critical strengths
  • The carefully controlled comparison provides useful evidence about how pretraining choices affect text embeddings.
  • Multiple model scales, diverse adaptation regimes, and broad evaluations make the study informative.
  • The main findings are clearly qualified by model scale and supervision budget, offering practical insights for the community.
Critical weaknesses
  • Each pretrained configuration uses only one seed. This substantially limits confidence in the central comparisons, particularly when diffusion’s gains over MLM are small (for example, 0.2502 versus 0.2435 on retrieval with 10k triplets at 500M.)
  • All experiments use one ModernBERT-style architecture family. This supports experimental control but leaves the generality of the conclusions uncertain.
  • The abstract is overloaded with acronyms, experimental details, and numerical results. It should first explain the research problem, motivation, and main takeaway in language accessible to readers outside text embedding research.
Questions for authors
  1. Can you repeat the key comparisons with multiple independent pretraining and adaptation seeds and report variability, prioritizing the 500M settings with limited supervision?
  2. What hardware and GPU-hours were required for pretraining, and what resources would these repetitions require?
  3. Which findings do you expect to generalize beyond this architecture family, and why?
1 comment

Thank you for highlighting the need for repeated runs. We now report both repeated adaptation and independent pretraining in Table~13 (provided in the updated manuscript, page 18, upper half).

Weakness 1 and Question 1. We reproduced all six original 500M SimCSE/10k retrieval results exactly. We then repeated adaptation with five seeds on the original pretrained models. Diffusion performs better than MLM in all five 10k comparisons: 0.2508±0.00170.2508\pm0.0017 versus 0.2414±0.00270.2414\pm0.0027. We also completed two new pretraining runs for each objective, giving three pretrained models per objective. Diffusion performs better than MLM after SimCSE and with 100k triplets in all three comparisons. Both bidirectional methods perform better than the causal model under SimCSE and 10k adaptation for every pretraining seed. The ranking of diffusion and MLM at 10k differs between pretrained models. The table reports standard deviations and separates the two types of repeated runs.

Question 2. The details are in Appendix A (provided in the updated manuscript, page~12, lower part). A successful new 500M/10B run takes about six hours on four H100 80GB GPUs, or 24 GPU-hours. At this rate, the six new runs require about 144 GPU-hours.

Weakness 2 and Question 3. Using one architecture keeps model design consistent across the comparison. MLM and diffusion also use the same bidirectional attention and mean pooling. We expect the amount of adaptation data to matter in other model families too: fine-tuning can change the pretrained representations and the ranking of methods. We added a short paragraph in Results to summarize the new checks and make the evidence easier to follow.

Weakness 3. We revised the opening of the abstract to state the practical problem: choosing a pretraining objective when labeled data for adaptation are limited. We also shortened the list of experimental details so that the research question is easier to follow.

Reviewer_sY4FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is an empirical paper that investigates the effect of three pretraining objectives on downstream embedding performance -- autoregressive LM, masked LM and masked-diffusion LM. The experiments suggests that there is no silver bullet when it comes to pretraining objectives and their effectiveness depends on the downstream use.

Critical strengths
  • The research question is interesting and appropriate for this conference.
  • I appreciate that the authors evaluated the models using multiple fine-tuning techniques; this really helps to better see the different strengths of autoregressive and masked language modeling. The same is true for the range of different downstream tasks.
Critical weaknesses

My main objection is the limited range of pretrained models, which is really troubling for a paper that is solely an empirical study of already established techniques. It is well known that masked (diffusion) language models are much less sample efficient (during pretraining) than autoregressive models. Thus, setting the pretraining budget to static 20 tokens per parameter, according to the Chinchilla scaling laws (that were established specifically for autoregressive models), looks naive and unfair to the bidirectional models. it is also unrealistic, no recent language models are trained with such a budget, and this study tells us nothing about language models at scale, trained on thousands of tokens per parameter.

I completely understand that it's difficult to scale up training to this extent because of compute constraints. Nevertheless, one can still establish scaling laws by extrapolating from smaller scales -- without requiring a bigger compute budget than what you have already used for pretraining.

Unfortunately, I cannot see how your empirical study would be helpful to me if I were to decide what pretraining objective to use for my new text embedding model. The study only tells me what objective to use if I constrain myself to 20 tokens per parameters -- which doesn't seem very helpful.

Questions for authors

It is quite popular to initialize masked (diffusion) LMs from an autoregressive LM, it's a simple way of overcoming the sample inefficiency. What do you think about this approach, based on the results you observed in your study?

2 comments

Thank you for raising this question. We added an experiment with ten times more pretraining tokens. The results are in Table 14(a) (provided in the updated manuscript, page 18, lower half).

Weakness 1. The original budget gives each objective the same amount of pretraining data. This allows a controlled comparison at a shared data budget. We have clarified this point in the pretraining section. We trained all three 60M models on 12B tokens, or about 200 tokens per parameter. The learning-rate schedules cover the full training period. Diffusion has the highest mean SimCSE retrieval score: 0.19790.1979, compared with 0.19010.1901 for MLM and 0.18330.1833 for the causal model. The diffusion and MLM means include three independent pretraining runs. The table reports their standard deviations. With full supervision, longer pretraining also improves all three methods by about 6-7 score points. These results show that diffusion remains a useful starting point for embeddings at a much larger pretraining budget.

Question 1. We see this as a promising approach, especially when labeled data for retrieval are limited. In our study, causal models perform strongly with full supervision, while bidirectional models give better retrieval with little supervision. This suggests that AR pretraining learns useful features, but more adaptation is needed to make them effective for retrieval. Starting from an AR checkpoint and continuing with masked/diffusion training could adapt these features using unlabeled text, reducing reliance on labeled pairs.

Thank you. You have responded to my main objection and ran the training on the bigger token budget. I think it's only fair that I increase my rating from 2 to 3 :) Please just make sure to put this result more visibly into the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.