acceptodds
Under review as a conference paper at ICLR 2027

Learning to Retrieve via Reinforcement Learning in Embedding Space

Abstract

Dense retrieval models are typically trained with contrastive objectives that learn effective representations but do not directly optimize retrieval metrics or downstream task performance. To address this problem, we introduce **RELER** (**RE**inforcement **LE**arning for **R**etrieval), a reinforcement learning framework that enables existing embedding models to *learn to retrieve* directly in embedding space and align to task-specific rewards. We train RELER by sampling unit-length query and document embedding actions from von Mises–Fisher (vMF) distributions centered on normalized encoder outputs, scoring the resulting retrieval or downstream outcomes as rewards, and updating the encoder with REINFORCE using a leave-one-out baseline (RLOO). As exploration in the high-dimensional embedding space is prone to sampling noise, we further propose conditional-mean projection (CMP), which projects each sampled embedding onto the low-dimensional subspace spanned by its encoder output and the candidate embeddings it is compared against, reducing noise in the policy gradient while preserving its expectation. We evaluate RELER on BRIGHT, a benchmark with reasoning-intensive queries that remain challenging for existing embedding models. RELER consistently outperforms InfoNCE and LambdaLoss in average nDCG@10 when post-training BGE-M3 and Qwen3-Embedding backbones. We further evaluate downstream utility through retrieval-augmented generation (RAG), where we adapt only the query encoder while keeping the document index and generator fixed. Across seven QA datasets, jointly optimizing retrieval and answer rewards improves both average retrieval performance and answer quality in RAG.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Dense Retrieval, Text Embedding Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vQkBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RELER, a reinforcement learning framework for training dense retrieval models by directly optimizing retrieval and downstream task rewards. RELER samples query and document embeddings from von Mises–Fisher (vMF) distributions centered on the encoder outputs. The sampled embeddings are evaluated using retrieval metrics, and the encoder is updated using REINFORCE with a leave-one-out baseline (RLOO). The paper introduces two techniques to improve training: (1) product rollouts, which reuse sampled query and document embeddings to obtain more reward evaluations, and (2) conditional-mean projection (CMP), which reduces policy gradient variance by removing random components that do not affect retrieval scores.

Experiments on BRIGHT show that RELER outperforms InfoNCE and LambdaLoss across three embedding backbones. The authors also evaluate RELER in a fixed-index RAG setting, where the query encoder is optimized using retrieval and answer-level rewards.

Critical strengths
  • The paper explores a different training objective for dense retrieval models by directly optimizing retrieval metrics rather than relying on contrastive learning. Treating embeddings as continuous actions is an interesting idea, and the framework can naturally incorporate non-differentiable downstream rewards.

  • CMP addresses an important issue with applying policy gradient methods in high-dimensional embedding spaces. The paper provides a theoretical justification for the projection and shows that it preserves the expected gradient under the stated assumptions. The reported 291–590× reduction in gradient variance is impressive.

Critical weaknesses
  • While RELER consistently improves average nDCG@10, the gains over InfoNCE are relatively small for the Qwen3-Embedding models. For example, on BRIGHT with original queries, Qwen3-Embedding-4B improves from 30.08 to 30.46, and Qwen3-Embedding-0.6B improves from 22.64 to 23.38. The improvements are also uneven across datasets, with noticeable performance drops on some subsets. This shows that the proposed method is not significantly stronger compared to simple InfoNCE-based learning.

  • The main retrieval experiments focus on BRIGHT, using fewer than 5K training queries from ReasonRank. Although BRIGHT is a challenging benchmark, it would be useful to evaluate whether the improvements transfer to other retrieval settings, such as standard BEIR datasets, and transfer to data-abundant scenario.

  • RAG improvements are relatively small, and the comparison is not fully controlled. In the fixed-index RAG experiments, the mixed reward improves average answer F1 from 40.76 to 41.10 over the pretrained model. Moreover, RELER uses online retrieval over the full index during training, while InfoNCE uses a fixed offline candidate pool. Therefore, it is difficult to determine how much of the improvement over InfoNCE comes from the RL objective rather than the different training candidate sets.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vQkB

We sincerely thank you for the time and effort you put into reviewing our paper and for your thoughtful comments. We respond to your comments point by point below.

W1: “The gains over InfoNCE are relatively small for the Qwen3-Embedding models … noticeable performance drops on some subsets.”

Response: RELER improves mean nDCG@10 over continued InfoNCE across all three backbones and both query settings (Table 1). Moving from original to GPT-4 reasoning queries, the gain increases from 0.38 to 0.78 points on Q3E-4B and from 1.86 to 3.22 on BGE-M3. The reasoning-query gains are consistent across seeds: RELER outperforms InfoNCE at every seed on all three backbones, and all three 95% paired t intervals are above zero (Tables 5 and 6).

The LeetCode drop is likely due to the data distribution mismatch, since the original ReasonRank paper also reports lower LeetCode scores after reranking, alongside gains in the overall BRIGHT average [1].

[1] ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability.


W2: “Transfer to other retrieval settings, such as standard BEIR datasets, and transfer to data-abundant scenario.”

Response: We evaluate retrieval beyond BRIGHT through 10 tasks in MTEB English v2 drawn from BEIR datasets and their updated variants. This evaluation covers retrieval datasets across different domains. Our new three-seed CL→CL / CL→RELER study evaluates the full 41-task suite across seven task types, including these 10 retrieval tasks. Starting from Qwen3-0.6B, we trained with InfoNCE for 1,200 steps on E2Rank, then continued each seed checkpoint with CL or RELER for another 1,200 steps, using the same data, learning rate, batch size, and a fresh optimizer/scheduler.

Training recipeTask averageRetrieval
Initial CL60.98 ± 0.0351.63 ± 0.10
CL→CL60.34 ± 0.0650.74 ± 0.08
CL→RELER61.34 ± 0.0650.92 ± 0.13

CL→RELER improves the task average by 1.00 point over continued CL and 0.36 over initial CL. Both continuation recipes show a retrieval drop from initial CL, possibly due to a distribution mismatch between E2Rank and the evaluation datasets. However, RELER achieves higher task and retrieval averages than CL under the same continuation settings, at every seed.

Our focus is directly optimizing ranking and downstream rewards through embedding actions, supported by CMP's analysis and controlled comparisons. Further analysis of this drop and systematic data scaling are beyond the scope of this paper.


W3: “Mixed reward improves average answer F1 from 40.76 to 41.10 … difficult to determine how much … comes from the RL objective rather than the different training candidate sets.”

Response: We have added a three-seed RAG study in the revision, covering all four training recipes (Tables 3 and 14). The document index and generator remain frozen. Seven-dataset averages are:

RecipeEMF1Hit@10
Pretrained32.3040.7655.87
InfoNCE31.10 ± 0.1539.46 ± 0.2451.77 ± 0.10
RLOO, mixed reward32.94 ± 0.1341.15 ± 0.1956.43 ± 0.08

Mixed reward improves average F1 by 1.68 points over InfoNCE and 0.38 over pretrained. EM, F1, and Hit@10 all improve over both references at every seed, showing that the gains recur across training runs.

Online retrieval lets RELER learn from the RAG outcomes of its current query policy. Each sampled query embedding searches the full frozen index. The retrieved top-10 passages are passed to the frozen generator, and the mixed reward evaluates both retrieval quality and the resulting answer. A fixed shortlist would provide no feedback on useful passages newly reached outside that pool as the policy adapts. For InfoNCE, offline hard-negative mining supplies the positive–negative comparisons used by the supervised contrastive loss. Although their effects are difficult to separate, reward optimization and online candidate access are both part of RELER, so gains from either contribute to the method.


We hope our responses address your concerns, and we look forward to further discussion.

Reviewer_3QUBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RELER (REinforcement LEarning for Retrieval), a reinforcement learning framework for post-training dense retrieval/embedding models. The core observation is that contrastive objectives (e.g., InfoNCE) optimize representation learning indirectly and do not directly target retrieval metrics (nDCG) or downstream RAG answer quality. RELER treats the normalized query/document embeddings as actions sampled from von Mises–Fisher (vMF) policies on the unit sphere, scores retrieval or downstream outcomes as rewards, and updates the encoder with REINFORCE using a leave-one-out baseline (RLOO). Two technical components are emphasized: (i) product rollouts (pairing all G query draws with all G document bundles for G² correlated rewards), and (ii) conditional-mean projection (CMP), a Rao–Blackwell-style variance reduction that projects sampled actions onto the subspace spanned by their policy mean and reward-relevant candidate directions, claimed to preserve the expected gradient while reducing noise.

Main results: On BRIGHT (reasoning-intensive retrieval), RELER outperforms InfoNCE and LambdaLoss in average nDCG@10 across BGE-M3, Qwen3-Embedding-0.6B, and -4B, using <5k training queries. In a fixed-index RAG setting (only the query encoder adapted; index and generator frozen), RLOO recipes with retrieval+answer rewards improve retrieval and answer quality over InfoNCE across seven QA datasets. An additional experiment training embeddings directly from a general-purpose LLM (Qwen3-0.6B) shows RELER does not beat InfoNCE on MTEB retrieval.

Critical strengths

Novel framing: Treating query/document embeddings themselves as RL actions sampled from vMF policies on the unit sphere rather than sampling discrete document selections or generated text is a genuinely distinct contribution relative to the cited prior work (ROPG-RL, Retrieval-GRPO, HARR, GRACE, Embed-RL, Bendada et al.).

Empirical rigor and honesty: Results use three paired seeds with reported means ± SD on BRIGHT; ablations isolate CMP, rollout design, reward granularity, pairwise weight λ, and alignment ρ. The gradient-variance study directly measures the claimed 291–590× variance reduction at fixed encoder states, and the authors disclose the 11–14% per-draw runtime cost. The paper candidly reports negative/mixed results: CMP is not used in RAG, RELER does not beat InfoNCE from a general-purpose LLM and subset-level trade-offs are acknowledged Practical relevance. The fixed-index query-only adaptation addresses a real deployment constraint (avoiding index rebuilds) and keeps deterministic single-vector inference at test time.

Critical weaknesses

Modest gains over the strongest baseline: On BRIGHT, RELER's advantage over InfoNCE is often small: e.g., Q3E-0.6B original queries 23.38 vs 22.64 (+0.74), Q3E-4B 30.46 vs 30.08 (+0.38), Q3E-4B GPT-4 queries 31.07 vs 30.29 (+0.78). The largest gains are on BGE-M3. Given that RELER's full recipe adds considerable machinery (product rollouts, CMP, pairwise reward), the cost-benefit tradeoff for the Qwen backbones is marginal, and the paper's own abstract claim of "consistently outperforms" should be weighed against these small margins. Reference baselines in Table 6 are far above RELER: The literature comparison shows ReasonEmbed-Q3E-4B (Redapter) at 37.1 average vs RELER's 30.46 on the same backbone family. The authors correctly note different protocols, but this raises the question of whether RELER's post-training gains are competitive with the current best reasoning-retrieval methods, or merely improve over a weak pretrained starting point. CMP's practical benefit is partly confounded with optimization: The paper states CMP "improves accuracy and reduces seed variation" (Table 2, 21.91→23.38) and shows large fixed-state variance reduction (Table 9). However, Appendix A.2/C.5 acknowledge the variance guarantee is per-term and does not order the summed gradient; and the key end-task improvement from CMP (+1.47 avg with combined reward) is attributed to "more effective optimization through reduced rollout noise" without a controlled isolation of the optimization effect from other sources. The link between measured fixed-state SNR and downstream accuracy remains correlational. Negative result in the LLM-from-scratch setting undercuts generality. Appendix E shows RELER does not beat InfoNCE on MTEB retrieval (50.04 vs 51.52) from Qwen3-0.6B. This suggests RELER's benefit may be confined to targeted post-training of already-strong embedding models on narrow, reasoning-intensive distributions, limiting broader claims about embedding-space RL. Single-seed limitations in secondary experiments: The fixed-index RAG experiments (Table 3, 11) and the LLM-to-embedding experiment (Table 12) use only seed 42. Given the small margins (e.g., mixed reward F1 41.10 vs pretrained 40.76; EM 32.96 vs 32.30), single-seed results make it hard to assess statistical reliability, especially since the authors themselves emphasize seed variability on BRIGHT.

Limited scope of downstream evaluation: RAG uses a single generator (Qwen2.5-7B-Instruct), a single corpus (wiki18), and only two training datasets (NQ, HotpotQA). Robustness to generator choice, context depth beyond top-10, and other corpora is not explored. Hit@10 interpretation. The authors acknowledge Hit@10 "ignores passage order, distracting context, and evidence removed by the generator's input limit" . Given this, the small answer-quality improvements over the pretrained encoder are hard to attribute confidently to better retrieval versus noise.

Questions for authors

Given the small RELER-over-InfoNCE margins on the Qwen backbones (+0.38 to +0.78 avg nDCG@10), are these differences statistically meaningful under your three-seed protocol? Could you report significance or confidence intervals on the paired differences? In the fixed-index RAG and LLM-from-scratch experiments, how much of the RELER/RLOO advantage (or InfoNCE advantage in Table 12) is attributable to online full-index retrieval versus the RL objective itself? Can you run a matched-candidate-access comparison to disentangle these? CMP is omitted in the fixed-index setting because projection "need not preserve the full-index top-k list." Is there an approximate CMP variant (e.g., over an enlarged candidate shortlist) that could recover variance reduction while bounding the bias? Did you measure any resulting bias empirically?

The combined reward improves EM/F1 only marginally over the pretrained encoder in RAG (Table 3). Could you provide multiple seeds for Tables 3, 11, and 12 to establish whether these gains are robust? How sensitive is RELER to the group size G=64? No ablation over G is reported, yet product rollouts generate G² reward cells, which has direct compute implications.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3QUB — Part 1/2

We sincerely thank you for the time and effort you put into reviewing our paper and for your thoughtful comments. We respond to your comments point by point below.

“Modest gains over the strongest baseline: …”

“Could you report significance or confidence intervals on the paired differences?”

Response: RELER outperforms InfoNCE in mean nDCG@10 across all six backbone/query conditions. With GPT-4 reasoning queries, the gain is positive at every seed on all three backbones. Table 6 reports RELER minus InfoNCE:

BackboneQueriesMean gain95% paired t interval
Q3E-0.6BOriginal†+0.733[−0.004, +1.470]
Q3E-4BOriginal†+0.380[−0.968, +1.728]
BGE-M3Original+1.863[+1.675, +2.051]
Q3E-0.6BGPT-4+0.635[+0.384, +0.885]
Q3E-4BGPT-4+0.782[+0.501, +1.062]
BGE-M3GPT-4+3.225[+2.896, +3.553]

Intervals are 95% paired t intervals across three training seeds on fixed evaluation sets (df=2). † Original-query Qwen intervals use per-seed inputs rounded to 0.01.


“How sensitive is RELER to the group size G=64? … product rollouts generate G² reward cells, which has direct compute implications.”

Response: We evaluated G=16–128 on Q3E-0.6B BRIGHT original queries over three seeds (Tables 11–12):

GnDCG@10Reward cells per record
1623.47 ± 0.34256
3223.43 ± 0.151,024
6423.38 ± 0.174,096
12823.46 ± 0.1116,384

Means span only 0.093 points. G=16 achieves comparable accuracy with 1/16 of G=64's reward cells; we retain G=64 for consistency with the main experiments.

BRIGHT training time remains similar to InfoNCE and G has small influence. Transformer forward/backward passes dominate training computation. RELER encodes texts once per batch and reuses the outputs for embedding sampling, product rollouts, reward computation, and CMP. Increasing G adds work on these vectors without repeating Transformer computation; G² counts reward evaluations, not additional encoder passes. Deployment uses deterministic single-vector retrieval with no rollout overhead.


“The literature comparison shows ReasonEmbed-Q3E-4B (Redapter) at 37.1 average vs RELER's 30.46 on the same backbone family.”

Response: ReasonEmbed and RELER use different training protocols, so their scores in Table 7 serve as literature references and do not constitute a fair comparison of the training methods. Our controlled experiments fix training settings, and RELER outperforms both InfoNCE and LambdaLoss in mean score across all six backbone/query conditions. These comparisons establish RELER's benefit as a post-training method.


“CMP is omitted in the fixed-index setting because projection "need not preserve the full-index top-k list."”

“Is there an approximate CMP variant … that could recover variance reduction while bounding the bias? Did you measure any resulting bias empirically?”

Response: The matched training ablation shows CMP's practical benefit: enabling it in both reward components raises BRIGHT from 21.91 ± 0.70 to 23.38 ± 0.17 (Table 2, rows a,b). Proposition 1 shows that CMP preserves the expected gradient and reduces the variance of each projected summand. Together, these results support more effective optimization through reduced rollout noise (Section 4.1.3).

Full-index RAG uses unprojected query-only RLOO. We have not evaluated approximate CMP or its bias. A shortlist-based extension would need to bound omitted-document effects and account for action-dependent candidate selection. This follows from CMP's requirement that the weighted gradient term depend on the retained information (Appendix A.2).

AuthorsOpenReview ↗

Response to Reviewer 3QUB — Part 2/2

“Negative result in the LLM-from-scratch setting undercuts generality.”

Response: RELER complements InfoNCE as a reward-based post-training stage. Our main BRIGHT experiments start from Qwen3-Embedding-0.6B/4B and BGE-M3; RELER has the highest mean in all six backbone/query conditions. In the new CL→RELER study, we first trained Qwen3-0.6B with InfoNCE for 1,200 E2Rank steps, then continued each seed checkpoint with CL or RELER for another 1,200 steps, using the same learning rate, batch size, and fresh optimizer/scheduler. RELER improves the 41-task average by 1.00 over continued CL and 0.36 over initial CL. Retrieval is 50.92 versus 50.74 for continued CL; both task and retrieval averages favor RELER at every seed. Both continuation scores are below initial CL's retrieval score of 51.63, possibly reflecting a training/evaluation distribution mismatch. Studying this drop in detail is beyond our scope.

Table 15 starts from Qwen3-0.6B without embedding-specific training. InfoNCE leads on retrieval at every seed: 51.63 ± 0.10 versus RELER's 50.39 ± 0.31. Its broader in-batch and cross-device negatives may help establish representations for large-corpus retrieval, while this RELER recipe uses own-query candidates.


“Single-seed limitations in secondary experiments: …”

“Hit@10 interpretation. … the small answer-quality improvements over the pretrained encoder are hard to attribute confidently to better retrieval versus noise.”

“Could you provide multiple seeds for Tables 3, 11, and 12 to establish whether these gains are robust?”

Response: We have added three-seed results for all four RAG recipes and both general-purpose initialization recipes. Seven-dataset RAG macro-averages are:

RecipeEMF1Hit@10
Pretrained32.3040.7655.87
InfoNCE31.10 ± 0.1539.46 ± 0.2451.77 ± 0.10
RLOO, graded nDCG32.15 ± 0.2640.54 ± 0.4156.24 ± 0.10
RLOO, answer F132.26 ± 0.0740.80 ± 0.0956.55 ± 0.09
RLOO, mixed32.94 ± 0.1341.15 ± 0.1956.43 ± 0.08

Datasets are averaged equally within each seed before computing mean ± SD. Pretrained is one deterministic reference. All trained recipes retain query-anchor coefficient 0.5; the index and generator are frozen. Table 14 gives per-dataset results.

Mixed reward improves EM, F1, and Hit@10 over both references at every seed. Its F1 gain over pretrained is +0.384, with 95% t interval [−0.097, +0.865]; the paired gain over InfoNCE is +1.683 [+1.354, +2.013]. These intervals summarize training-seed variation on fixed evaluation sets (n=3, df=2).

With the index and generator frozen and decoding greedy, query adaptation changes the supplied contexts. EM/F1 directly evaluate the generated answers from the ordered, truncated contexts. Hit@10 measures answer containment in any of the ten retrieved passages. The new study shows consistent gains in both answer quality and retrieval hits across training runs.


“Can you run a matched-candidate-access comparison to disentangle these?”

Response: We use online retrieval so that rewards evaluate the current query policy in the full RAG system. Each sampled embedding retrieves its own top-10 passages from the frozen index; retrieval and answer rewards then guide the update. A fixed shortlist would provide no feedback on useful passages newly reached outside that pool as the policy adapts. For InfoNCE, offline hard-negative mining supplies the positive–negative comparisons used by the supervised contrastive loss. A matched-candidate-access experiment has not been run. Although their effects are difficult to separate, reward optimization and online candidate access are both part of RELER, so gains from either contribute to the method.


We hope our responses address your concerns, and we look forward to further discussion.

Reviewer_nBHMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a RELER, new approach to training dense retrieval models. Usually retrieval models are trained with a contrastive loss between query and document embeddings, which is not directly optimized for the downstream metrics like nDCG. To optimize on them directly, the authors used RL in which embedding are the actions and reward is the retrieval metric.mTo sample actions, ie, embeddings, they used von Mises–Fisher distribution around encoder output. From this distribution they sample query and document embeddings, compute similarity and rank them, and use ranking score as the reward. The encoder is updated with RLOO.

Because the exploration happens in high dimensional space, it is prone to sampling noise. The authors solution is conditional-mean projection (CMP), where they project each sample onto the low dimensional space before computing gradient, reducing this noise without biasing the gradient. The authors show that this reduced gradient variance by ~300-600x.

The authors showed that RELER outperformed InfoNCE and learning to rank LambdaLoss in average nDCG@10 on BRIGHT dataset.

Critical strengths
  • The approach is well thought out. Exploring in embedding space and using conditional-mean projection to remove noise while preserving the signal, which is empirically supported by substantially reducing gradient variance.

  • The improvement is consistent - RELER has the best average score across 3 backbones and 2 query setting.

Critical weaknesses
  • In Table 1 we see a drop in Leetcode performance. The authors mentioned the variation in subsets is partly because of how closely subset matches training data distribution. Why not add a regularizer?
Questions for authors
  • Did you explore doing RELER/RL as post training after contrastive training?
3 comments
AuthorsOpenReview ↗

Response to Reviewer nBHM

We sincerely thank you for the time and effort you put into reviewing our paper and for your thoughtful comments. We respond to your comments point by point below.

W1: “In Table 1 we see a drop in Leetcode performance. … Why not add a regularizer?”

Response: The LeetCode drop is likely due to the data distribution mismatch. The original ReasonRank paper also reports lower LeetCode scores after reranking, alongside gains in the overall BRIGHT average [1]. Regularization may help mitigate this drop. Given that the concern is specific to LeetCode, we do not plan to introduce a regularizer across all training data. Studying regularization for such distribution shifts is beyond the scope of this paper.

[1] ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability.


Q1: “Did you explore doing RELER/RL as post training after contrastive training?”

Response: Yes. RL post-training of pretrained embedding models is our main setting. Our BRIGHT experiments start from Qwen3-Embedding-0.6B/4B and BGE-M3, comparing RELER with continued InfoNCE from the same checkpoints (Section 4.1; Table 1).

We also tested an explicit CL→RL sequence starting from Qwen3-0.6B on E2Rank. After CL training, we continued each of three seed checkpoints with either InfoNCE (CL→CL) or RELER (CL→RELER) for another 1200 steps, using the same settings. Evaluation covers all 41 MTEB English v2 tasks across seven task types, including 10 retrieval tasks.

Training recipeTask averageRetrieval
Initial CL60.98 ± 0.0351.63 ± 0.10
CL→CL60.34 ± 0.0650.74 ± 0.08
CL→RELER61.34 ± 0.0650.92 ± 0.13

CL→RELER improves the task average by 1.00 point over continued CL and 0.36 over initial CL. Both continuation recipes show a retrieval drop from initial CL, possibly due to a distribution mismatch between E2Rank and the evaluation datasets; studying this drop in detail is beyond the scope of this paper. RELER achieves higher task and retrieval averages than CL under the same continuation settings, at every seed.


We hope our responses address your concerns, and we look forward to further discussion.

Thanks for the response. Its harder to judge when retrieval performance is sometimes lower compared to baseline. Why not evaluate on held out dataset so that distribution mismatch is not a concern? Showing that it works for out of distribution would make this a stronger paper.

I will keep my Rating 3.

Thank you for the follow-up. We agree that evaluating on a held-out set drawn from the training distribution would help clarify whether we see similar drops relative to the baseline even within the same distribution. Regarding BRIGHT, the training data was constructed by ReasonRank specifically for this benchmark; therefore, we have not altered it, nor does it represent an out-of-distribution scenario. The performance decline on LeetCode has also been observed in other studies, suggesting this may be a common issue rather than one unique to our work. Addressing the performance drop on this specific subset of BRIGHT falls outside the scope of this paper. As for MTEBv2, it should be noted that the same E2Rank dataset was used in both stages (as shown in the table provided in the response); this reuse of the same dataset may be one of the factors contributing to the decline in retrieval performance. Given that these are mixed datasets and MTEB is a widely adopted evaluation benchmark, we opted for this particular experimental setup. A separate out-of-distribution evaluation would then give a clearer picture of how well the method generalizes. However, it is worth emphasizing that under identical experimental conditions, RELER consistently outperforms CL. We will clarify these limitations in the paper and temper our claims accordingly.

Comments on the paper

2 comments

Dear Reviewers,

As the rebuttal period is drawing to a close, we would appreciate any feedback on our responses, particularly whether they address your concerns and whether any questions remain. We would be happy to provide further clarification before the deadline.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

Dear Area Chair and Reviewers,

We thank the reviewers for their thoughtful feedback and their recognition of the following strengths.

  • Embedding-space reinforcement learning. RELER treats query and document embeddings as continuous actions, so a retriever can learn directly from ranking or answer quality (3QUB, vQkB) while retaining single-vector retrieval at inference (3QUB).
  • Lower gradient variance. Conditional-mean projection removes noise from directions that leave ranking scores unchanged while preserving the expected policy gradient (nBHM, vQkB).
  • Consistent average gains on BRIGHT. RELER achieved higher mean nDCG@10 than both InfoNCE and LambdaLoss across all three backbones and both query settings (nBHM).

During the rebuttal, we addressed three questions raised by the reviewers.

  • Reliability and comparison settings. We examined how consistently the retrieval and answer-quality gains held across training runs and clarified how each RAG training method accessed candidate documents (3QUB, vQkB).
  • Post-training after contrastive learning. Starting from the same contrastively trained checkpoints, switching to RELER yielded a higher average across MTEB tasks than continuing with InfoNCE (nBHM).
  • Sampling cost. A smaller sampling budget achieved comparable retrieval accuracy in the BRIGHT setting tested (3QUB).

We appreciate the reviewers’ feedback on how the gains vary across settings. In weighing these results, we hope the reviewers and the Area Chair will give particular weight to the methodological contribution of RELER. We believe that treating embeddings themselves as actions in reinforcement learning opens a new direction for training embedding models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.