acceptodds
Under review as a conference paper at ICLR 2027

A Study of Downstream Rewards for Retrieval-only Adaptation

Abstract

Reward design is a central challenge in learning multi-step retrieval policies from downstream feedback. We investigate how different reward signals enable retrieval-only adaptation, where a compact embedding model learns to select evidence while the answer-generating language model remains frozen. Using Q-RAG as a common framework, we compare rewards based on generated answers, reference likelihoods, and semantic uncertainty, with supporting-fact supervision as a reference. We also introduce Group Information Gain, a contrastive likelihood reward over answer candidates evaluated offline. Experiments on HotpotQA and 2WikiMultiHopQA characterize how these signals affect evidence retrieval and answer quality. Using the selected reward, we then train the retrieval policy over a Wikipedia corpus containing approximately 21 million passages and evaluate it on seven QA datasets. Results show that retriever-only fine-tuning improves downstream answer quality and achieves competitive performance with fine-tuned LLM-based search policies while updating only a compact embedding model. Adapting the retriever already required by the RAG pipeline thus offers a flexible way to specialize search while leaving the LLM unchanged for other tasks.

Primary area:
reinforcement learning
Keywords:
RAG, retrieval, reinforcement learning, QA, multi-hop RAG

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5tDfOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies downstream reward design for retrieval-only adaptation, where a compact retrieval policy is trained while the answer-generating LLM remains frozen, using Q-RAG as a common framework.

It compares answer-based, likelihood-based, and uncertainty-based rewards against supporting-fact supervision, and introduces Group Information Gain, an offline contrastive likelihood reward over judged answer candidates.

Experiments on multi-hop QA show that retrieval quality and answer quality are not the same objective, uncertainty-derived rewards do not beat exact match on the evaluated benchmarks, and cost does not track answer quality. Using the selected LLM-as-a-Judge reward, the authors train on a Wikipedia corpus of about 21M passages and evaluate on seven QA datasets, showing that retriever-only fine-tuning improves downstream answer quality competitively while updating only a compact embedding model.

Critical strengths

A controlled, systematic comparison of downstream rewards under a fixed retrieval framework. The paper holds the Q-RAG pipeline and the Answerer LLM fixed while training only the retriever, then compares answer-based, likelihood-based, and uncertainty-based rewards against supporting-fact supervision. This is consequential because reward design is the paper’s central question, and the controlled setup makes it credible to attribute differences to the reward signal rather than to unrelated pipeline changes. It also yields useful negative and comparative findings—e.g., reward contrastiveness does not reliably predict post-training answer quality, uncertainty-based rewards do not beat Exact Match on the evaluated benchmarks, and cost does not track answer quality. These findings materially support acceptance as a technically sound empirical study, even if the paper is not a breakthrough method paper.

Practical validation of retrieval-only adaptation beyond the fixed-pool setting. Using the selected LLM-as-a-Judge reward, the authors train the retrieval policy over a Wikipedia corpus of roughly 21M passages and evaluate on seven QA datasets, plus a multi-turn dialogue QA setting. This is consequential because it shows that the central idea is not confined to a small fixed-pool benchmark: retriever-only fine-tuning can improve downstream answer quality while updating only a compact embedding model and leaving the LLM unchanged. This practical demonstration substantially strengthens the paper’s relevance and supports acceptance.

Critical weaknesses

The empirical evidence lacks sufficient external validity to support the paper’s central comparative claims. The main reward comparison is limited to a single model family and a limited set of dataset configurations, primarily two fixed-pool datasets, while several conclusions depend on differences of only a few points or on fine-grained reward rankings. The authors acknowledge the limited experimental coverage in the limitations section, but they do not provide validation across model families. Although a second judge (Qwen3-32B) is reported at evaluation time, it remains within the same model family and does not rule out family-specific bias. In addition, all reward feedback comes from the same Qwen3-4B family, with the Judge and Answerer being the same model during training, so judge–answerer error correlation may bias the training signal. This is consequential because the paper’s core contribution is a comparative recommendation about which downstream reward to use. If the ranking holds only under a specific model family and a limited set of datasets, the practical guidance is not robustly established in terms of external validity. At minimum, the authors should scope their claims to the current setting and provide a small-scale cross-family or independent-judge check.

The evaluation protocol and the Full-Wiki validation are not clean enough to substantiate the practical recommendation.
Checkpoint selection uses 1,000 HotpotQA dev questions that are also part of the final evaluation set; the authors note this but do not correct it, so reported answer quality may be optimistically biased, especially when reward differences are small. Moreover, the Full-Wiki setting updates only the state encoder while the fixed-pool comparison trains both encoders, so the practical validation is not a clean extension of the main reward comparison. I do not treat the Full-Wiki setup difference alone as a rejection-level flaw, since a full-scale action-encoder training may be impractical; however, the paper should either justify this difference explicitly as a practical demonstration rather than a direct extension, or provide a matched small-scale control. Together with the checkpoint-selection issue, this weakens the evidential link between the controlled comparison and the practical recommendation. These issues require clarification and, if unresolved, weaken the practical claim.

Taken together, these two weaknesses do not invalidate the paper’s controlled comparison, which is a genuine strength. However, they leave the central comparative recommendation and the practical validation insufficiently supported in the current form. I lean toward weak rejection, but would be willing to reconsider if the authors appropriately scope their claims and provide a held-out checkpoint-selection split plus at least a small-scale cross-family or independent-judge validation.

Questions for authors
  1. External validity of the reward ranking. The main reward comparison appears to be limited to a single model family and a limited set of dataset configurations, primarily two fixed-pool datasets, and several conclusions depend on small differences between rewards. Can the authors provide at least a small-scale validation of the central reward rankings—for example, Answer F1 and Retrieval F1 on HotpotQA and 2Wiki—using an independent judge model or a different Answerer model family? If not, can they clarify which conclusions should be treated as robust beyond the specific Qwen3-4B setting and which are merely suggestive?

  2. Checkpoint selection contamination. The paper states that checkpoints are selected on 1,000 HotpotQA dev questions that are also part of the final evaluation set. Can the authors rerun the main comparisons with a held-out checkpoint-selection split, or otherwise quantify how much this selection protocol could inflate the reported answer quality, especially for closely ranked rewards?

  3. Consistency of the Full-Wiki validation with the main comparison. In the fixed-pool setting both encoders are trained, whereas in Full-Wiki only the state encoder is updated. Can the authors clarify why this difference is necessary and whether the Full-Wiki results should be interpreted as a direct extension of the main reward comparison or as a separate practical demonstration? An explicit justification and limitation statement, or a matched small-scale control, would materially affect how strongly the practical recommendation can be supported.

3 comments
AuthorsOpenReview ↗

Answer to Reviewer 5tDf [p1]

We thank the reviewer for their careful reading of our manuscript and for their thoughtful comments and questions. We address these points below.

Weakness #1 and Question #1

Stability of the reward ranking

We tested whether the ranking of training rewards changes with the retriever’s training seed or with the model used to generate and judge answers. We trained Q-RAG on HotpotQA with two seeds for each of six rewards, using Qwen3-4B during training. For each checkpoint, Qwen3-4B and Gemma4-E4B-it each generated and judged answers from the same retrieved passages on the test runs. Values are means ± sample standard deviation across the four seed-to-model combinations.

Training rewardAnswer EM (%)Answer F1 (%)Judged accuracy (%)
LLM-as-a-Judge58.23 ± 0.9772.08 ± 0.3076.75 ± 2.78
Group IG57.75 ± 1.1371.65 ± 0.7076.03 ± 2.97
Exact Match57.15 ± 0.9970.78 ± 0.3675.05 ± 2.68
Synthetic Semantic Information Gain (SSIG)56.53 ± 1.1270.15 ± 0.2574.48 ± 2.80
Information Gain (IG)55.13 ± 0.5168.70 ± 0.2373.08 ± 1.88
Semantic Entropy Probes (SEP)51.10 ± 3.9764.18 ± 4.4068.23 ± 4.86

Ranking the six rewards separately for each of the three metrics in each of the four configurations gave 12 rankings. Nine reproduced the full row order shown above. The other three each differed by one adjacent swap, all under Gemma. The same two rewards therefore held the top two places in all 12 rankings. LLM-as-a-Judge ranked first in 10, including all four Answer EM rankings. Group IG performed better than IG on all 12 rankings.

Switching from Qwen to Gemma while holding the metric fixed gave a mean Spearman rank correlation of 0.971 across the matched comparisons (range 0.943–1.000). Switching the training seed while holding the evaluation model and metric fixed gave the same mean and range. Additionally we measured agreement with Kendall’s coefficient of concordance[1], which equals 1 for identical rankings. Across the 12 rankings, W = 0.975.

[1] Legendre, Pierre. "Species associations: the Kendall coefficient of concordance revisited." Journal of agricultural, biological, and environmental statistics 10.2 (2005): 226-245.

Cross-model validation and training

The previous experiment varied the model family only at evaluation. Here we also varied which model generated training answers and which model judged them, retraining both the state and action encoders in each configuration. For each checkpoint, Qwen3-4B and Gemma4-E4B-it answered from the same retrieved passages. For cross-model training, we focused on the three best performing rewards: LLM-as-a-Judge, Group IG, Exact Match. Each cell reports Answer EM / Answer F1 (%). Bold scores mark the best result for each evaluation answerer.

Training rewardTraining answererTraining judgeEvaluation answererAnswer EM (%)Answer F1 (%)
LLM-as-a-JudgeGemmaGemmaQwen58.372.7
LLM-as-a-JudgeGemmaGemmaGemma55.969.6
LLM-as-a-JudgeQwenGemmaQwen58.673.5
LLM-as-a-JudgeQwenGemmaGemma54.568.2
LLM-as-a-JudgeGemmaQwenQwen58.472.8
LLM-as-a-JudgeGemmaQwenGemma56.469.8
Group IGQwen—Qwen57.572.1
Group IGQwen—Gemma53.967.3
Exact MatchGemma—Qwen56.971.2
Exact MatchGemma—Gemma54.667.8

With either evaluation answerer, the highest EM and F1 came from training with that answerer and a judge from the other family. All three Judge configurations outscored Group IG and Exact Match on F1 with both answerers. Qwen outscored Gemma at evaluation by ~3. points in Answer EM and Answer F1. With Gemma as the training answerer, switching the training judge from Gemma to Qwen increased F1 by only 0.1 point when Qwen answered and 0.2 when Gemma answered. LLM-as-a-Judge retained its lead on HotpotQA under cross-family scoring.

We plan to evaluate and add all remaining rewards, and extend these experiments to 2WikiMultiHopQA in the final version of the paper.

AuthorsOpenReview ↗

Answer to Reviewer 5tDf [p2]

Weakness 2 and Question 2

Full-Wiki checkpoint selection

We thank the reviewer for identifying the checkpoint-selection overlap. The Qwen3-4B Judge checkpoint in Table 2 was selected on 1,000 HotpotQA development questions included in the final evaluation. To avoid checkpoint-selection bias from the overlapping evaluation questions, we used the final checkpoint saved after 30k optimizer updates. This training budget was fixed in advance. Both the stopping point and the checkpoint used for evaluation were therefore independent of performance on the 1,000 HotpotQA questions used for the original checkpoint selection. Additionaly we want to point out that The Qwen2.5-3B result in Table 2 already used the final checkpoint. Its best checkpoint used a disjoint training holdout of 500 HotpotQA and 500 2Wiki questions.

The table reports Answer EM (%). Each named LLM supplies training feedback and evaluation answers. For each Judge run, best (contaminated data for Qwen3 based checkpoint) and last use the retrieval settings of its Table 2 entry. Avg. is the equal-weight mean across seven datasets. The rows marked † reproduce Table 2.

MethodNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
Q-RAG (Qwen2.5-3B Judge), best30.0850.4337.5432.0931.4610.0521.6030.46
Q-RAG (Qwen2.5-3B Judge), last†30.3350.0037.1132.1031.789.8126.4031.08
Q-RAG (Qwen2.5-3B Group IG), last26.5146.8334.7927.6221.327.2013.6025.41
Q-RAG (Qwen3-4B Judge), best†33.4157.3139.8537.7432.2911.6328.8034.43
Q-RAG (Qwen3-4B Judge), last33.0757.6940.1937.5333.6011.9628.8034.69
Q-RAG (Qwen3-4B Group IG), last31.8656.0138.7733.1117.9310.9223.2030.26

Surprisingly, the last checkpoints slightly outperform those selected as best in overall Full-Wiki performance: +0.26 points for Qwen3-4B and +0.61 points for Qwen2.5-3B. We argue that this outcome is plausible for three reasons. First, the 1,000 overlapping selection questions used for Qwen3-4B constitute only 1.93% of the 51,713 evaluation questions and 13.5% of HotpotQA. Second, the evaluation suite is heterogeneous: HotpotQA carries one seventh of the macro-average weight, while the other six datasets include both single-hop and multi-hop tasks. A checkpoint selected on this small HotpotQA subset therefore need not be optimal across the full suite. Third, the gains may indicate that continued training to the fixed 30,000-step endpoint is more beneficial for overall performance than selecting an earlier checkpoint based on a lucky fluctuation on a small subset that does not fully reflect the test distribution.

Additionaly we also report final Group IG checkpoints. The newly added Full-Wiki results show a substantially larger gap in Answer EM between Group IG and LLM-as-a-Judge than in Table 1 and our other experiments. To investigate this difference, we added an analysis of the behavior of retrievers trained with these two rewards in the Full-Wiki setting in Appendix F.

AuthorsOpenReview ↗

Answer to Reviewer 5tDf [p3]

Question 3

State only encoder training on Full-Wiki

Our original fixed-pool experiments followed Q-RAG’s default configuration, which trains both encoders. We adopted state-only training for computational reasons in Full-Wiki. Updating the action encoder in Full-Wiki would require repeatedly re-encoding 21 million passages and refreshing the vector index. Asynchronous refreshes would additionally require coordinating index updates and handling stale passage embeddings. We therefore train only the state encoder and reuse the fixed corpus index. We agree that Full-Wiki serves as a practical demonstration of reward-driven retrieval adaptation in a different setting.

To isolate the effect of freezing the action encoder, we additionally ran a matched control comparing state-only training with training both encoders. The control uses the HotpotQA fixed-pool setting with Qwen3-4B as the Answerer. For LLM-as-a-Judge, Gemma4-E4B-it provides the training reward. All scores are percentages.

Training rewardTrainable encodersRetrieval EMRetrieval F1Answer EMAnswer F1Qwen3-4B judgeQwen3-32B judge
LLM-as-a-JudgeState + action77.487.558.673.575.882.3
LLM-as-a-JudgeState only77.387.458.473.275.682.1
Support FactsState + action89.193.759.474.376.783.3
Support FactsState only88.493.359.074.076.583.2

Freezing the action encoder decreases Answer F1 by 0.3 percentage points for both rewards. Answer EM decreases by 0.2 points for LLM-as-a-Judge and 0.4 points for Support Facts. Across all six metrics, the largest change is 0.7 points, so every difference is below one percentage point. Support Facts retains its 0.8-point advantage in Answer F1 in both training modes. The additional control shows that freezing the action encoder has only a small effect in the tested fixed-pool configurations.

Reviewer_tdJSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates downstream reward design for retrieval-only reinforcement learning using the existing Q-RAG framework. The authors systematically compare answer-based, likelihood-based, and uncertainty-based rewards, and introduce Group Information Gain (Group IG), a contrastive extension of single-reference likelihood feedback. The experiments show that LLM-as-a-Judge achieves the strongest downstream answer quality among the evaluated rewards, while Group IG consistently improves over conventional Information Gain. The paper also examines the relationships between evidence retrieval quality, reward contrastiveness, and downstream answer performance. Finally, the authors evaluate judge-based training in a Full-Wiki environment containing approximately 21 million passages, demonstrating competitive performance against selected LLM-based search baselines.

Critical strengths
  1. Systematic reward comparison. The paper evaluates several representative reward formulations under a shared Q-RAG framework, providing a relatively controlled empirical comparison of their effects on retrieval and downstream answer quality.
  2. Consistent improvement over single-reference IG. The proposed Group IG improves over conventional Information Gain across the evaluated datasets, demonstrating the potential value of contrasting correct and incorrect answer candidates.
Critical weaknesses
  1. Insufficient analysis of reward properties and learning dynamics. The paper reports several interesting observations, including the underperformance of uncertainty-based rewards and the discrepancy between reward contrastiveness and downstream policy performance. However, these findings receive limited analysis in terms of reward distributions, optimization behavior, and alignment with the final task objective. In particular, it remains unclear why Group IG achieves stronger evidence contrastiveness but lower downstream answer quality than LLM-as-a-Judge. This limits the explanatory value of the study beyond reporting performance differences.
  2. Limited experimental coverage. The main reward comparisons are conducted on only two datasets (HotpotQA and 2WikiMultiHopQA), using a single retriever backbone for each dataset and a fixed Qwen3-4B Answerer. Although the paper includes additional Full-Wiki experiments, these evaluate only the selected LLM-as-a-Judge reward rather than comparing different reward formulations. Consequently, the limited diversity of datasets and model configurations makes it difficult to assess whether the observed reward rankings and conclusions hold more broadly.
Questions for authors
  1. Reward mechanisms: Why does Group IG achieve stronger evidence contrastiveness but lower downstream answer quality than LLM-as-a-Judge? Can the authors provide further analysis to explain this discrepancy?
  2. Retrieval budget: How sensitive are the different reward formulations to the number of retrieval steps? Since Q-RAG uses a fixed retrieval budget, could varying the number of steps affect the relative performance rankings of the evaluated rewards?
3 comments

We thank the reviewer for their time and valuable feedback and questions.

Weakness 1 and Question 1

We expanded our analysis and conducted additional experiments to address two questions: why Group IG’s stronger contrastiveness does not translate into the best answer quality, and what gives LLM-as-a-Judge an advantage over Group IG despite its lower contrastiveness.

1. Why contrastiveness does not guarantee answer quality

Evidence contrastiveness measures how often a reward prefers the full annotated supporting context to one containing no annotated supporting passages. We examined its limits on 300 randomly sampled HotpotQA development questions with ten-passage pools and a fixed Qwen3-4B Answerer.

Non-gold passages can improve the answer. Here, non-gold means passages outside the annotated supporting set. Let τ\tau denote the original trajectory and τrepl\tau_{\mathrm{repl}} the trajectory with all non-gold passages replaced by alternatives from the same question’s pool, keeping gold passages and context size fixed. For each metric mm (Answer EM/Answer F1), we compute Δm=m(τ)−m(τrepl)\Delta m = m(\tau) - m(\tau_{\mathrm{repl}}). The table reports this difference in percentage points, averaged over all admissible replacements per question.

Training rewardMean Δ Answer EM, p.p.Mean Δ Answer F1, p.p.
LLM-as-a-Judge+1.29+0.96
EM+4.55+6.32
IG+0.06−0.08
Group IG+4.72+4.56
All trained policies except IG learned to select useful non-gold passages. For example, for a question about GZA’s 2005 collaboration with DJ Muggs, all four policies retrieve the non-gold Grandmasters (album) passage, which directly supplies the answer. Removing it changes Answer EM from 1 to 0 for all four policies.

The benefit of supporting facts varies across questions. Providing all supporting facts does not always enable the LLM to answer correctly. Conversely, some questions can be answered without context using knowledge stored in the model’s parameters.

We compared answers generated without context and with all annotated supporting facts. By exact match (EM), 46.33% of questions were answered correctly only with supporting context (0→1), 40.00% in neither condition (0→0), 11.67% in both, and 2.00% only without context. The table focuses on the 0→0 group and reports Retrieval EM (both gold passages retrieved) and the policy’s Answer EM.

Training reward0→0: Retrieval EM0→0: Answer EM
LLM-as-a-Judge70.834.17
EM71.673.33
IG69.175.00
Group IG84.173.33

In this group, Group IG has much higher Retrieval EM than LLM-as-a-Judge (84.17% vs. 70.83%), while its Answer EM is lower (3.33% vs. 4.17%). More frequent retrieval of the annotated evidence therefore does not translate into higher answer accuracy here.

2. What helps explain Judge’s stronger downstream performance

Smaller gap to reward-maximizing trajectories. On the same 300 questions, we scored 45 candidate retrieval trajectories under each reward, one for each pair of passages, preserving their order in the candidate pool. We also included the trajectory produced by the trained policy if it was absent from this set. We compared Answer EM for the policy’s trajectory and the reward-maximizing trajectory, averaging over ties.

Training rewardPolicy Answer EMReward-maximizing context Answer EMGap [95% CI], p.p.
LLM-as-a-Judge51.3360.529.19 [5.11, 13.48]
EM52.6772.3319.67 [15.33, 24.33]
IG51.3365.4314.09 [9.70, 18.57]
Group IG53.3361.338.00 [4.00, 12.00]

Scores are percentages here. Judge and Group IG have smaller gaps to their respective reward-maximizing trajectories (9.19 and 8.00 points) than EM and IG (19.67 and 14.09 points). Their learned policies therefore come closer to the answer quality obtained by exhaustive selection under their own rewards.

Answer quality when retrieval is incomplete. The following partition covers all 7,405 HotpotQA dev questions using two retrained policies with a Qwen3-4B Answerer. Judge uses a Gemma training judge, and Group IG follows the Table 1 recipe. Contributions weight each group’s F1 gap by its share.

Both gold passages retrieved byShareJudge Answer F1Group IG Answer F1Contribution to overall gap, p.p.
Both policies67.3%78.578.6−0.0
Judge only10.1%77.847.0+3.1
Group IG only10.9%60.275.4−1.7
Neither policy11.7%53.750.1+0.4
Overall100%73.571.7+1.8

When only Judge retrieves both gold passages, its Answer F1 advantage is 30.8 points. In the reverse case, Group IG leads by only 15.2 points. This is consistent with Judge retaining more useful partial or alternative evidence, or omitting gold passages that the Answerer does not need.

An additional difference in the Full-Wiki setting. We updated the Full-Wiki results in Table 2 and added Group IG. LLM-as-a-Judge’s advantage over Group IG in average answer quality is larger here than in the Table 1 experiments. We added Appendix F to analyse this gap and the retrievers’ behavior in this setting. In Full-Wiki, all questions retrieve from the same corpus of approximately 21 million passages, allowing the same articles to recur as hubs across unrelated queries (Hubness can also arise naturally in high-dimensional embedding spaces[1] ).

Appendix F shows that both trained retrievers concentrate on recurring articles more than pretrained GTE. Each retriever’s most frequent article occurs in 24.28% (Group IG), 7.83% (Judge), and 0.08% (GTE zero-shot) of questions (Fig. 5a). Trajectories containing more hub passages tend to yield lower average answer quality under both rewards (Fig. 5b), while Group IG retrieves hubs much more frequently than LLM-as-a-Judge (Fig. 5a). Together with the reward probe in Appendix F, this suggests a possible explanation for Group IG’s lower answer quality: hubs may receive high reward by suppressing the likelihood of incorrect answer candidates while providing little relevant evidence. This can encourage the retriever to select hubs at the expense of useful supporting passages. The weaker attraction to hubs under LLM-as-a-Judge may help explain its stronger downstream performance.

References

[1] Radovanović, M., Nanopoulos, A., and Ivanović, M. (2010). Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data. Journal of Machine Learning Research, 11, 2487–2531.

Weaknes 2: Coverage.

Second model family. We varied which model generated training answers and which model judged them, retraining both the state and action encoders in each configuration. For each checkpoint, Qwen3-4B and Gemma4-E4B-it answered from the same retrieved passages. For cross-model training, we focused on the three best performing rewards: LLM-as-a-Judge, Group IG, Exact Match. Each cell reports Answer EM / Answer F1 (%). Bold scores mark the best result for each evaluation answerer.

Training rewardTraining answererTraining judgeEvaluation answererAnswer EM (%)Answer F1 (%)
LLM-as-a-JudgeGemmaGemmaQwen58.372.7
LLM-as-a-JudgeGemmaGemmaGemma55.969.6
LLM-as-a-JudgeQwenGemmaQwen58.673.5
LLM-as-a-JudgeQwenGemmaGemma54.568.2
LLM-as-a-JudgeGemmaQwenQwen58.472.8
LLM-as-a-JudgeGemmaQwenGemma56.469.8
Group IGQwen—Qwen57.572.1
Group IGQwen—Gemma53.967.3
Exact MatchGemma—Qwen56.971.2
Exact MatchGemma—Gemma54.667.8

With either evaluation answerer, the highest EM and F1 came from training with that answerer and a judge from the other family. All three Judge configurations outscored Group IG and Exact Match on F1 with both answerers. Qwen outscored Gemma at evaluation by ~3. points in Answer EM and Answer F1. With Gemma as the training answerer, switching the training judge from Gemma to Qwen increased F1 by only 0.1 point when Qwen answered and 0.2 when Gemma answered. LLM-as-a-Judge retained its lead on HotpotQA under cross-family scoring.

We plan to evaluate and add all remaining rewards, and extend these experiments to 2WikiMultiHopQA in the final version of the paper.

Full-Wiki with two further backbones.

Answerer and judgeRetrieverRewardAvg. EMGain over untrained, p.p.
Qwen3-4BGTEJudge34.69+6.2
Qwen3-4BGTEGroup IG30.26+2.3
Qwen3-4Bbge-m3Judge33.77+4.9
Qwen2.5-3BGTEJudge31.08+5.4
Qwen2.5-3BGTEGroup IG25.41−0.3
Qwen2.5-3BQwen3-EmbeddingJudge31.19+5.9

Results use the last saved checkpoint of the corresponding training run. Gains are computed against the corresponding untrained retriever with the same answerer and retrieval settings.

All rows use the Full-Wiki environment of Section 4.3: retrieval over the 21 million passages of Wiki-18, six retrieval steps, and the 51,713 questions of the seven Search-R1 evaluation sets, of which only HotpotQA and 2Wiki were seen during training. The score is exact match against the accepted answer aliases, averaged with equal weight over the seven sets. Each row trains the retriever with the reward named in the third column and reads the retrieved passages with the Answerer named in the first column, which also serves as the training-time judge.

Third dataset. The revised paper now includes MuSiQue dataset (Appendix G. Table 6). The ordering of the judge and Group IG rewards is the same as on 2Wiki, with wider difference.

Q2. Retrieval budget.

We did not vary T in isolation, but the three settings we report span budgets from two to six steps with the same rewards, and the ordering does not change. On HotpotQA (T = 2, ten candidate paragraphs, two gold) the judge and Group IG are within seed variation. On MuSiQue (T = 4, twenty candidate paragraphs, up to four gold; Table 1) the judge is ahead, so the likelihood rewards fall behind as the budget and the pool grow. In Full-Wiki (T = 6, open corpus) the judge is ahead of Group IG. Budget and pool size change together across these settings, so we cannot attribute the widening gap to the number of steps alone, but a larger budget never reverses the ranking. For the judge-trained Full-Wiki policy, Appendix D shows that horizons between six and ten steps change the result by less than one point. The final version will add a direct comparison on HotpotQA with T = 3 and T = 4 for Exact Match, LLM-as-a-Judge, and Group IG, where only the budget changes.

Reviewer_AkEcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides a comprehensive study on retrieval-only adaption in multi-hop RAG and evaluates various downstream reward formulations, inlucindg rewards based on generated answers, reference likehoods, semantic uncertinty within a unified retrieval framework. The paper also proposes Group Information Gain (Group IG) as an improved reward formulation. Extensive experiments on multi-hop QA datasets and analysis provide some interesting findings.

Critical strengths
  1. This paper as an emperical analysis paper has a potentially interesting contribution: it investigates how different reward formulations affect multi-step retrieval training, and whether the quality of a reward can be predicted before training. Some interesting findings, including that uncertainty-derived rewards do not reach the quality of the simplest string-matching signal, and contrastiveness did not reliably predict answer quality after training, might provide insights to the comminuty.
  2. The paper is generally well-writing and easy to follow. The extensive experiments seem solid with controlled comparison, making the findings reliable and non-trivial. The conclusion also provide some meaningful insights.
Critical weaknesses
  1. The paper primarily builds upon an existing Q-RAG framework and compares various reward formulations, while keeping the rest of the framework unchanged. It offers comparative observations but limited mechanistic understanding of the relationship between reward formulation, retrieval behavior, and downstream QA performance. The paper can be improved if not just observations but also the deeper mechanisms why these rewards produce these outcomes. For example, why LM-as-a-Judge achieved the strongest answer quality among the evaluated downstream rewards?
  2. The paper draws conclusions about reward formulation in multi-step RAG, but the evidence may be restricted to one particular training framework (Q-RAG) and generator configuration. Experiments involving additional retrieval frameworks or LLMs would help establish whether the findings generalize.
Questions for authors

Why LLM-as-a-Judge achieved the strongest answer quality among the evaluated downstream rewards? Do you have deeper analysis or explanations beyond the findings?

2 comments
AuthorsOpenReview ↗

Answer to Reviewer AkEc [p1]

Thank you for your careful review and constructive feedback.

Weakness 1 and Question 1

We expanded our analysis and conducted additional experiments to address two questions: why Group IG’s stronger contrastiveness does not translate into the best answer quality, and what gives LLM-as-a-Judge an advantage over Group IG despite its lower contrastiveness.

1. Why contrastiveness does not guarantee answer quality

Evidence contrastiveness measures how often a reward prefers the full annotated supporting context to one containing no annotated supporting passages. We examined its limits on 300 randomly sampled HotpotQA development questions with ten-passage pools and a fixed Qwen3-4B Answerer.

Non-gold passages can improve the answer. Here, non-gold means passages outside the annotated supporting set. Let τ\tau denote the original trajectory and τrepl\tau_{\mathrm{repl}} the trajectory with all non-gold passages replaced by alternatives from the same question’s pool, keeping gold passages and context size fixed. For each metric mm (Answer EM/Answer F1), we compute Δm=m(τ)−m(τrepl)\Delta m = m(\tau) - m(\tau_{\mathrm{repl}}). The table reports this difference in percentage points, averaged over all admissible replacements per question.

Training rewardMean Δ Answer EM, p.p.Mean Δ Answer F1, p.p.
LLM-as-a-Judge+1.29+0.96
EM+4.55+6.32
IG+0.06−0.08
Group IG+4.72+4.56
All trained policies except IG learned to select useful non-gold passages. For example, for a question about GZA’s 2005 collaboration with DJ Muggs, all four policies retrieve the non-gold Grandmasters (album) passage, which directly supplies the answer. Removing it changes Answer EM from 1 to 0 for all four policies.

The benefit of supporting facts varies across questions. Providing all supporting facts does not always enable the LLM to answer correctly. Conversely, some questions can be answered without context using knowledge stored in the model’s parameters.

We compared answers generated without context and with all annotated supporting facts. By exact match (EM), 46.33% of questions were answered correctly only with supporting context (0→1), 40.00% in neither condition (0→0), 11.67% in both, and 2.00% only without context. The table focuses on the 0→0 group and reports Retrieval EM (both gold passages retrieved) and the policy’s Answer EM.

Training reward0→0: Retrieval EM0→0: Answer EM
LLM-as-a-Judge70.834.17
EM71.673.33
IG69.175.00
Group IG84.173.33

In this group, Group IG has much higher Retrieval EM than LLM-as-a-Judge (84.17% vs. 70.83%), while its Answer EM is lower (3.33% vs. 4.17%). More frequent retrieval of the annotated evidence therefore does not translate into higher answer accuracy here.

2. What helps explain Judge’s stronger downstream performance

Smaller gap to reward-maximizing trajectories. On the same 300 questions, we scored 45 candidate retrieval trajectories under each reward, one for each pair of passages, preserving their order in the candidate pool. We also included the trajectory produced by the trained policy if it was absent from this set. We compared Answer EM for the policy’s trajectory and the reward-maximizing trajectory, averaging over ties.

Training rewardPolicy Answer EMReward-maximizing context Answer EMGap [95% CI], p.p.
LLM-as-a-Judge51.3360.529.19 [5.11, 13.48]
EM52.6772.3319.67 [15.33, 24.33]
IG51.3365.4314.09 [9.70, 18.57]
Group IG53.3361.338.00 [4.00, 12.00]

Scores are percentages here. Judge and Group IG have smaller gaps to their respective reward-maximizing trajectories (9.19 and 8.00 points) than EM and IG (19.67 and 14.09 points). Their learned policies therefore come closer to the answer quality obtained by exhaustive selection under their own rewards.

Answer quality when retrieval is incomplete. The following partition covers all 7,405 HotpotQA dev questions using two retrained policies with a Qwen3-4B Answerer. Judge uses a Gemma training judge, and Group IG follows the Table 1 recipe. Contributions weight each group’s F1 gap by its share.

Both gold passages retrieved byShareJudge Answer F1Group IG Answer F1Contribution to overall gap, p.p.
Both policies67.3%78.578.6−0.0
Judge only10.1%77.847.0+3.1
Group IG only10.9%60.275.4−1.7
Neither policy11.7%53.750.1+0.4
Overall100%73.571.7+1.8

When only Judge retrieves both gold passages, its Answer F1 advantage is 30.8 points. In the reverse case, Group IG leads by only 15.2 points. This is consistent with Judge retaining more useful partial or alternative evidence, or omitting gold passages that the Answerer does not need.

AuthorsOpenReview ↗

Answer to Reviewer AkEc [p1]

An additional difference in the Full-Wiki setting. We updated the Full-Wiki results in Table 2 and added Group IG. LLM-as-a-Judge’s advantage over Group IG in average answer quality is larger here than in the Table 1 experiments. We added Appendix F to analyse this gap and the retrievers’ behavior in this setting. In Full-Wiki, all questions retrieve from the same corpus of approximately 21 million passages, allowing the same articles to recur as hubs across unrelated queries (Hubness can also arise naturally in high-dimensional embedding spaces[1] ).

Appendix F shows that both trained retrievers concentrate on recurring articles more than pretrained GTE. Each retriever’s most frequent article occurs in 24.28% (Group IG), 7.83% (Judge), and 0.08% (GTE zero-shot) of questions (Fig. 5a). Trajectories containing more hub passages tend to yield lower average answer quality under both rewards (Fig. 5b), while Group IG retrieves hubs much more frequently than LLM-as-a-Judge (Fig. 5a). Together with the reward probe in Appendix F, this suggests a possible explanation for Group IG’s lower answer quality: hubs may receive high reward by suppressing the likelihood of incorrect answer candidates while providing little relevant evidence. This can encourage the retriever to select hubs at the expense of useful supporting passages. The weaker attraction to hubs under LLM-as-a-Judge may help explain its stronger downstream performance.

Weakness 2: Generalization across LLM families

To test generalization beyond the original LLM configuration, we added HotpotQA experiments with Qwen3-4B and Gemma4-E4B-it, varying the model family during both evaluation and training. Q-RAG remains fixed to isolate reward effects.

Cross-family evaluation. We trained Q-RAG with two seeds for each of six rewards, using Qwen3-4B during training. For each checkpoint, Qwen3-4B and Gemma4-E4B-it each generated and judged answers from the same retrieval trajectories. Values are means ± sample standard deviation across the four seed-to-model combinations.

Training rewardAnswer EM (%)Answer F1 (%)Judged accuracy (%)
LLM-as-a-Judge58.23 ± 0.9772.08 ± 0.3076.75 ± 2.78
Group IG57.75 ± 1.1371.65 ± 0.7076.03 ± 2.97
Exact Match57.15 ± 0.9970.78 ± 0.3675.05 ± 2.68
Synthetic Semantic Information Gain (SSIG)56.53 ± 1.1270.15 ± 0.2574.48 ± 2.80
Information Gain (IG)55.13 ± 0.5168.70 ± 0.2373.08 ± 1.88
Semantic Entropy Probes (SEP)51.10 ± 3.9764.18 ± 4.4068.23 ± 4.86

Judge and Group IG take the top two places in all 12 rankings (two seeds × two evaluation models × three metrics). Judge ranks first in ten, including all four EM rankings, and Group IG exceeds IG in all 12. Changing only the evaluation model gives a mean Spearman correlation of 0.971 (range 0.943–1.000).

Cross-family training. We retrained both state and action encoders with the following answerer–judge combinations and evaluated each checkpoint with both answerers using the same retrieval trajectories. Qwen and Gemma denote Qwen3-4B and Gemma4-E4B-it. Bold scores mark the best result for each evaluation answerer.

Training rewardTraining answererTraining judgeEvaluation answererAnswer EM (%)Answer F1 (%)
LLM-as-a-JudgeGemmaGemmaQwen58.372.7
LLM-as-a-JudgeGemmaGemmaGemma55.969.6
LLM-as-a-JudgeQwenGemmaQwen58.673.5
LLM-as-a-JudgeQwenGemmaGemma54.568.2
LLM-as-a-JudgeGemmaQwenQwen58.472.8
LLM-as-a-JudgeGemmaQwenGemma56.469.8
Group IGQwen—Qwen57.572.1
Group IGQwen—Gemma53.967.3
Exact MatchGemma—Qwen56.971.2
Exact MatchGemma—Gemma54.667.8

For both answerers, the highest EM and F1 come from training with that answerer and a judge from the other family. All three Judge configurations outperform Group IG and Exact Match on F1 with both answerers. This supports transfer across the tested LLM families within Q-RAG. We plan to extend cross-family training to all remaining rewards and to 2WikiMultiHopQA in the final version.

References

[1] Radovanović, M., Nanopoulos, A., and Ivanović, M. (2010). Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data. Journal of Machine Learning Research, 11, 2487–2531.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.