- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the question of whether diffusion pretraining improves downstream performance for language models adapted for information retrieval via contrastive fine tuning. The authors perform a broad experimental analysis of the question, considering matched parameter size, chinchilla optimally trained models trained with different pretraining objectives (MLM, causal LM, diffusion) as base models for subsequent IR fine tuning. The authors find that the diffusion objective is advantageous for larger-scale language models that are adapted for IR in a data limited regime, but that the advantage deteriorates with fine tuning dataset size.
Critical strengths
- The problem of pretraining recipe for downstream IR fine tuning is interesting and relevant to the field.
- The breadth of the paper's empirical experiments, from pretrianing configurations, to fine-tuning configuration, to evaluation breadth, is unusually strong.
Critical weaknesses
- The abstract itself does not frame a problem, but rather, presents a laundry list of findings. The paper would benefit from a clear problem framing within which the findings are situated. The introduction itself does this, so much of the framing regarding pretraining for IR models can likely be applied in the abstract.
- The largest weakness of the paper is that all experiments use a consistent pretraining corpus, which makes it difficult to generalize the findings to models trained with other pretraining corpora. The paper could be made much stronger if different pretraining corpora were considered.
Questions for authors
N/A
Thank you for suggesting a second corpus and clearer problem framing.
Weakness 2. We additionally trained all three 150M models from scratch on 3B FineWeb-Edu tokens. We kept the tokenizer, model settings, token budget, optimizer settings, and adaptation procedure the same. Table 14(b) (provided in the updated manuscript, page 18, lower half) shows that diffusion also performs better than MLM on this second corpus: 0.2511 versus 0.2352 with 100k triplets, and 0.2590 versus 0.2441 with full supervision. Appendix B (provided in the updated manuscript, page 13, lower part) describes the data preparation. The added paragraph in Results summarizes these findings.
Weakness 1. We revised the opening of the abstract to state the practical problem: choosing a pretraining objective when labeled data for adaptation are limited. We also shortened the list of experimental details so that the research question is easier to follow.