- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RELER, a reinforcement learning framework for training dense retrieval models by directly optimizing retrieval and downstream task rewards. RELER samples query and document embeddings from von Mises–Fisher (vMF) distributions centered on the encoder outputs. The sampled embeddings are evaluated using retrieval metrics, and the encoder is updated using REINFORCE with a leave-one-out baseline (RLOO). The paper introduces two techniques to improve training: (1) product rollouts, which reuse sampled query and document embeddings to obtain more reward evaluations, and (2) conditional-mean projection (CMP), which reduces policy gradient variance by removing random components that do not affect retrieval scores.
Experiments on BRIGHT show that RELER outperforms InfoNCE and LambdaLoss across three embedding backbones. The authors also evaluate RELER in a fixed-index RAG setting, where the query encoder is optimized using retrieval and answer-level rewards.
Critical strengths
-
The paper explores a different training objective for dense retrieval models by directly optimizing retrieval metrics rather than relying on contrastive learning. Treating embeddings as continuous actions is an interesting idea, and the framework can naturally incorporate non-differentiable downstream rewards.
-
CMP addresses an important issue with applying policy gradient methods in high-dimensional embedding spaces. The paper provides a theoretical justification for the projection and shows that it preserves the expected gradient under the stated assumptions. The reported 291–590× reduction in gradient variance is impressive.
Critical weaknesses
-
While RELER consistently improves average nDCG@10, the gains over InfoNCE are relatively small for the Qwen3-Embedding models. For example, on BRIGHT with original queries, Qwen3-Embedding-4B improves from 30.08 to 30.46, and Qwen3-Embedding-0.6B improves from 22.64 to 23.38. The improvements are also uneven across datasets, with noticeable performance drops on some subsets. This shows that the proposed method is not significantly stronger compared to simple InfoNCE-based learning.
-
The main retrieval experiments focus on BRIGHT, using fewer than 5K training queries from ReasonRank. Although BRIGHT is a challenging benchmark, it would be useful to evaluate whether the improvements transfer to other retrieval settings, such as standard BEIR datasets, and transfer to data-abundant scenario.
-
RAG improvements are relatively small, and the comparison is not fully controlled. In the fixed-index RAG experiments, the mixed reward improves average answer F1 from 40.76 to 41.10 over the pretrained model. Moreover, RELER uses online retrieval over the full index during training, while InfoNCE uses a fixed offline candidate pool. Therefore, it is difficult to determine how much of the improvement over InfoNCE comes from the RL objective rather than the different training candidate sets.
Questions for authors
See Weaknesses.
Response to Reviewer vQkB
We sincerely thank you for the time and effort you put into reviewing our paper and for your thoughtful comments. We respond to your comments point by point below.
Response: RELER improves mean nDCG@10 over continued InfoNCE across all three backbones and both query settings (Table 1). Moving from original to GPT-4 reasoning queries, the gain increases from 0.38 to 0.78 points on Q3E-4B and from 1.86 to 3.22 on BGE-M3. The reasoning-query gains are consistent across seeds: RELER outperforms InfoNCE at every seed on all three backbones, and all three 95% paired t intervals are above zero (Tables 5 and 6).
The LeetCode drop is likely due to the data distribution mismatch, since the original ReasonRank paper also reports lower LeetCode scores after reranking, alongside gains in the overall BRIGHT average [1].
[1] ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability.
Response: We evaluate retrieval beyond BRIGHT through 10 tasks in MTEB English v2 drawn from BEIR datasets and their updated variants. This evaluation covers retrieval datasets across different domains. Our new three-seed CL→CL / CL→RELER study evaluates the full 41-task suite across seven task types, including these 10 retrieval tasks. Starting from Qwen3-0.6B, we trained with InfoNCE for 1,200 steps on E2Rank, then continued each seed checkpoint with CL or RELER for another 1,200 steps, using the same data, learning rate, batch size, and a fresh optimizer/scheduler.
CL→RELER improves the task average by 1.00 point over continued CL and 0.36 over initial CL. Both continuation recipes show a retrieval drop from initial CL, possibly due to a distribution mismatch between E2Rank and the evaluation datasets. However, RELER achieves higher task and retrieval averages than CL under the same continuation settings, at every seed.
Our focus is directly optimizing ranking and downstream rewards through embedding actions, supported by CMP's analysis and controlled comparisons. Further analysis of this drop and systematic data scaling are beyond the scope of this paper.
Response: We have added a three-seed RAG study in the revision, covering all four training recipes (Tables 3 and 14). The document index and generator remain frozen. Seven-dataset averages are:
Mixed reward improves average F1 by 1.68 points over InfoNCE and 0.38 over pretrained. EM, F1, and Hit@10 all improve over both references at every seed, showing that the gains recur across training runs.
Online retrieval lets RELER learn from the RAG outcomes of its current query policy. Each sampled query embedding searches the full frozen index. The retrieved top-10 passages are passed to the frozen generator, and the mixed reward evaluates both retrieval quality and the resulting answer. A fixed shortlist would provide no feedback on useful passages newly reached outside that pool as the policy adapts. For InfoNCE, offline hard-negative mining supplies the positive–negative comparisons used by the supervised contrastive loss. Although their effects are difficult to separate, reward optimization and online candidate access are both part of RELER, so gains from either contribute to the method.
We hope our responses address your concerns, and we look forward to further discussion.