- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Revela+, extending Revela to train dense retrievers through pre-training, supervised fine-tuning, and preference tuning. It uses document prediction, answer prediction, and answer preference objectives to train the retriever through LM attention weights. Experiments on BRIGHT and BEIR show stage-wise retrieval improvements across five encoder backbones.
Critical strengths
- S1: The paper presents a promising extension of Revela by incorporating SFT and preference tuning into retriever training. Experiments show improvements over Revela and consistent stage-wise gains across multiple encoder backbones on BRIGHT and BEIR.
Critical weaknesses
- W1: The motivation for extending Revela to a three-stage pipeline is insufficiently developed. While answer prediction and preference comparison provide plausible supervision, the paper does not clearly explain what limitation of the preceding stage each addresses or why this progression is preferable to directly exploiting relevance supervision.
- W2: The empirical contribution relative to existing methods is insufficiently established. ReasonEmbed [1], also initialized from pretrained LLMs, achieves stronger BRIGHT results with contrastive learning but is omitted from the comparisons. It remains unclear what additional advantage the proposed training pipeline offers over existing approaches.
[1] Jianlyu Chen, Junwei Lan, Chaofan Li, Defu Lian, and Zheng Liu. 2026. ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1203–1221, San Diego, California, United States. Association for Computational Linguistics.
Questions for authors
None
4 comments
Thank you for the clarification and for the ReasonEmbed comparison. I agree that training data quality may explain much of the gap, but ReasonEmbed reports a higher BRIGHT score than Revela+ even with standard InfoNCE. How would Revela+ perform if trained on the same ReasonEmbed relevance pairs? This comparison would help clarify whether the gains over InfoNCE observed on your training data also hold on ReasonEmbed’s data.
We thank the reviewer for the follow-up and the suggestion. Training Revela+ on ReasonEmbed’s relevance pairs is not a drop-in change: our signal is trained on sequences rather than query–document pairs, so the data must first be adapted, which involves design choices that deserve careful study. This is not feasible to do properly within the remaining rebuttal period.
We note that Revela+ already trains on data curated by other groups specifically for contrastive training: ReasonIR and RaDeR were each trained on their own authors’ data and recipes, and Revela+ trained on these datasets outperforms both on BRIGHT (+7.7 and +6.1 points). ReasonEmbed builds on the same recipe as these baselines, LLM-synthesized reasoning-intensive queries with contrastive training, with more refined data curation. We will investigate training Revela+ on ReasonEmbed’s data and discuss it in the camera-ready version.
Thank you for the clarification. Whether the observed advantage over standard InfoNCE carries over to ReasonEmbed’s data remains an open question. My concern in W2 therefore remains, and I will keep my original rating. I view the paper as borderline and would not object to acceptance.
We thank the reviewer for their careful reading and for recognising Revela+ as a promising extension of Revela with consistent stage-wise gains across encoders and benchmarks. We address each weakness below.
Motivation for the three-stage pipeline
We agree the motivation should be stated more explicitly, and we have revised the introduction accordingly.
What each stage addresses
Pre-training needs no labels, but its retriever never scores documents from a query. In a blinded LLM-judged analysis of 70 BRIGHT-Pro queries, a quarter of its top-10 documents are off-topic. SFT addresses this by training the retriever on the documents that help the language model answer a query: the off-topic share halves and supporting documents rise from 39% to 47%. SFT, however, rewards any document that makes the answer more likely, and it never contrasts the relevant document with a plausible alternative. Preference tuning adds this contrast by comparing answers grounded in the relevant and in a mined negative document. Supporting documents rise further, to 50%, and the comparison adds 1.9 points on BRIGHT beyond continued answer prediction on the same data. Each stage also depends on the previous one: SFT from the base models reaches 20.1 NDCG@10, against 29.8 from the pre-trained pair.
Why not direct relevance supervision
We use the same relevance pairs, but through the language model’s loss, which measures how much a document helps produce the answer. With identical data and updates, encoder-only InfoNCE reaches 25.7 against 29.8 NDCG@10, and it falls below the pre-trained starting point on Biology and Earth Science, where it loses the pre-trained preference for complete passages that the language-model objective preserves (Appendix F.2). The three stages follow the same reasoning as language-model training, where each stage supplies a signal the previous one lacks (Ouyang et al., 2022).
We have stated this motivation in the introduction and added the analysis to Appendix H.
Comparison with stronger baselines (ReasonEmbed)
We have evaluated ReasonEmbed under our harness and have added this comparison (Section 4.2, Appendix D.2).
Our main contribution is a training signal applied across three training stages, and we control the training data. ReasonEmbed’s gains come largely from data. It synthesizes 81.7K training queries from BRIGHT’s own corpora as said in their paper (“we directly leverage the 12 datasets included in BRIGHT”), with BRIGHT’s task instructions, covering all 12 evaluation domains, and its authors acknowledge that their study is “currently limited to domains covered by existing benchmarks, particularly BRIGHT.” Its own ablation reflects this: its proposed loss adds only +1.3 points on BRIGHT over standard InfoNCE on the same data. We instead train on ReasonIR’s (Shao et al., 2025) varied-length (VL) split, whose queries and documents are LLM-generated from brainstormed task types and not drawn from BRIGHT’s domains, and we deliberately exclude ReasonIR’s HQ split, which is seeded from BRIGHT documents. Even so, without any training data (queries or documents) synthesized from BRIGHT’s corpora, Revela+ trails ReasonEmbed on the same Llama-3.1-8B backbone by only 4.0 points (32.1 vs. 36.1) and outperforms it on Earth Science (56.4 vs. 56.0), LeetCode (32.1 vs. 28.8) and AoPS (13.4 vs. 9.1). The fair comparisons for our signal are on shared data. With matched data, batches and updates, our signal outperforms InfoNCE by +4.1 points (29.8 vs. 25.7), and Revela+-8B outperforms ReasonIR and RaDeR (Das et al., 2025), the contrastive methods whose data we use, by +7.7 and +6.1 points (32.1 vs. 24.4 and 26.0).
We hope these clarifications and the additional analyses address the reviewer’s concerns, and we would be glad to answer any further questions.
References
Jianlyu Chen, Junwei Lan, Chaofan Li, Defu Lian, and Zheng Liu. 2026. ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1203–1221.
Debrup Das, Sam O’Nuallain, and Razieh Rahimi. 2025. RaDeR: Reasoning-aware Dense Retrieval Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19970–19997.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022).
Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, and Luke Zettlemoyer. 2025. ReasonIR: Training Retrievers for Reasoning Tasks. In Second Conference on Language Modeling (COLM 2025).