- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies downstream reward design for retrieval-only adaptation, where a compact retrieval policy is trained while the answer-generating LLM remains frozen, using Q-RAG as a common framework.
It compares answer-based, likelihood-based, and uncertainty-based rewards against supporting-fact supervision, and introduces Group Information Gain, an offline contrastive likelihood reward over judged answer candidates.
Experiments on multi-hop QA show that retrieval quality and answer quality are not the same objective, uncertainty-derived rewards do not beat exact match on the evaluated benchmarks, and cost does not track answer quality. Using the selected LLM-as-a-Judge reward, the authors train on a Wikipedia corpus of about 21M passages and evaluate on seven QA datasets, showing that retriever-only fine-tuning improves downstream answer quality competitively while updating only a compact embedding model.
Critical strengths
A controlled, systematic comparison of downstream rewards under a fixed retrieval framework. The paper holds the Q-RAG pipeline and the Answerer LLM fixed while training only the retriever, then compares answer-based, likelihood-based, and uncertainty-based rewards against supporting-fact supervision. This is consequential because reward design is the paper’s central question, and the controlled setup makes it credible to attribute differences to the reward signal rather than to unrelated pipeline changes. It also yields useful negative and comparative findings—e.g., reward contrastiveness does not reliably predict post-training answer quality, uncertainty-based rewards do not beat Exact Match on the evaluated benchmarks, and cost does not track answer quality. These findings materially support acceptance as a technically sound empirical study, even if the paper is not a breakthrough method paper.
Practical validation of retrieval-only adaptation beyond the fixed-pool setting. Using the selected LLM-as-a-Judge reward, the authors train the retrieval policy over a Wikipedia corpus of roughly 21M passages and evaluate on seven QA datasets, plus a multi-turn dialogue QA setting. This is consequential because it shows that the central idea is not confined to a small fixed-pool benchmark: retriever-only fine-tuning can improve downstream answer quality while updating only a compact embedding model and leaving the LLM unchanged. This practical demonstration substantially strengthens the paper’s relevance and supports acceptance.
Critical weaknesses
The empirical evidence lacks sufficient external validity to support the paper’s central comparative claims. The main reward comparison is limited to a single model family and a limited set of dataset configurations, primarily two fixed-pool datasets, while several conclusions depend on differences of only a few points or on fine-grained reward rankings. The authors acknowledge the limited experimental coverage in the limitations section, but they do not provide validation across model families. Although a second judge (Qwen3-32B) is reported at evaluation time, it remains within the same model family and does not rule out family-specific bias. In addition, all reward feedback comes from the same Qwen3-4B family, with the Judge and Answerer being the same model during training, so judge–answerer error correlation may bias the training signal. This is consequential because the paper’s core contribution is a comparative recommendation about which downstream reward to use. If the ranking holds only under a specific model family and a limited set of datasets, the practical guidance is not robustly established in terms of external validity. At minimum, the authors should scope their claims to the current setting and provide a small-scale cross-family or independent-judge check.
The evaluation protocol and the Full-Wiki validation are not clean enough to substantiate the practical recommendation.
Checkpoint selection uses 1,000 HotpotQA dev questions that are also part of the final evaluation set; the authors note this but do not correct it, so reported answer quality may be optimistically biased, especially when reward differences are small. Moreover, the Full-Wiki setting updates only the state encoder while the fixed-pool comparison trains both encoders, so the practical validation is not a clean extension of the main reward comparison. I do not treat the Full-Wiki setup difference alone as a rejection-level flaw, since a full-scale action-encoder training may be impractical; however, the paper should either justify this difference explicitly as a practical demonstration rather than a direct extension, or provide a matched small-scale control. Together with the checkpoint-selection issue, this weakens the evidential link between the controlled comparison and the practical recommendation. These issues require clarification and, if unresolved, weaken the practical claim.
Taken together, these two weaknesses do not invalidate the paper’s controlled comparison, which is a genuine strength. However, they leave the central comparative recommendation and the practical validation insufficiently supported in the current form. I lean toward weak rejection, but would be willing to reconsider if the authors appropriately scope their claims and provide a held-out checkpoint-selection split plus at least a small-scale cross-family or independent-judge validation.
Questions for authors
-
External validity of the reward ranking. The main reward comparison appears to be limited to a single model family and a limited set of dataset configurations, primarily two fixed-pool datasets, and several conclusions depend on small differences between rewards. Can the authors provide at least a small-scale validation of the central reward rankings—for example, Answer F1 and Retrieval F1 on HotpotQA and 2Wiki—using an independent judge model or a different Answerer model family? If not, can they clarify which conclusions should be treated as robust beyond the specific Qwen3-4B setting and which are merely suggestive?
-
Checkpoint selection contamination. The paper states that checkpoints are selected on 1,000 HotpotQA dev questions that are also part of the final evaluation set. Can the authors rerun the main comparisons with a held-out checkpoint-selection split, or otherwise quantify how much this selection protocol could inflate the reported answer quality, especially for closely ranked rewards?
-
Consistency of the Full-Wiki validation with the main comparison. In the fixed-pool setting both encoders are trained, whereas in Full-Wiki only the state encoder is updated. Can the authors clarify why this difference is necessary and whether the Full-Wiki results should be interpreted as a direct extension of the main reward comparison or as a separate practical demonstration? An explicit justification and limitation statement, or a matched small-scale control, would materially affect how strongly the practical recommendation can be supported.
3 comments
Answer to Reviewer 5tDf [p2]
Weakness 2 and Question 2
Full-Wiki checkpoint selection
We thank the reviewer for identifying the checkpoint-selection overlap. The Qwen3-4B Judge checkpoint in Table 2 was selected on 1,000 HotpotQA development questions included in the final evaluation. To avoid checkpoint-selection bias from the overlapping evaluation questions, we used the final checkpoint saved after 30k optimizer updates. This training budget was fixed in advance. Both the stopping point and the checkpoint used for evaluation were therefore independent of performance on the 1,000 HotpotQA questions used for the original checkpoint selection. Additionaly we want to point out that The Qwen2.5-3B result in Table 2 already used the final checkpoint. Its best checkpoint used a disjoint training holdout of 500 HotpotQA and 500 2Wiki questions.
The table reports Answer EM (%). Each named LLM supplies training feedback and evaluation answers. For each Judge run, best (contaminated data for Qwen3 based checkpoint) and last use the retrieval settings of its Table 2 entry. Avg. is the equal-weight mean across seven datasets. The rows marked † reproduce Table 2.
| Method | NQ | TriviaQA | PopQA | HotpotQA | 2Wiki | MuSiQue | Bamboogle | Avg. |
|---|---|---|---|---|---|---|---|---|
| Q-RAG (Qwen2.5-3B Judge), best | 30.08 | 50.43 | 37.54 | 32.09 | 31.46 | 10.05 | 21.60 | 30.46 |
| Q-RAG (Qwen2.5-3B Judge), last† | 30.33 | 50.00 | 37.11 | 32.10 | 31.78 | 9.81 | 26.40 | 31.08 |
| Q-RAG (Qwen2.5-3B Group IG), last | 26.51 | 46.83 | 34.79 | 27.62 | 21.32 | 7.20 | 13.60 | 25.41 |
| Q-RAG (Qwen3-4B Judge), best† | 33.41 | 57.31 | 39.85 | 37.74 | 32.29 | 11.63 | 28.80 | 34.43 |
| Q-RAG (Qwen3-4B Judge), last | 33.07 | 57.69 | 40.19 | 37.53 | 33.60 | 11.96 | 28.80 | 34.69 |
| Q-RAG (Qwen3-4B Group IG), last | 31.86 | 56.01 | 38.77 | 33.11 | 17.93 | 10.92 | 23.20 | 30.26 |
Surprisingly, the last checkpoints slightly outperform those selected as best in overall Full-Wiki performance: +0.26 points for Qwen3-4B and +0.61 points for Qwen2.5-3B. We argue that this outcome is plausible for three reasons. First, the 1,000 overlapping selection questions used for Qwen3-4B constitute only 1.93% of the 51,713 evaluation questions and 13.5% of HotpotQA. Second, the evaluation suite is heterogeneous: HotpotQA carries one seventh of the macro-average weight, while the other six datasets include both single-hop and multi-hop tasks. A checkpoint selected on this small HotpotQA subset therefore need not be optimal across the full suite. Third, the gains may indicate that continued training to the fixed 30,000-step endpoint is more beneficial for overall performance than selecting an earlier checkpoint based on a lucky fluctuation on a small subset that does not fully reflect the test distribution.
Additionaly we also report final Group IG checkpoints. The newly added Full-Wiki results show a substantially larger gap in Answer EM between Group IG and LLM-as-a-Judge than in Table 1 and our other experiments. To investigate this difference, we added an analysis of the behavior of retrievers trained with these two rewards in the Full-Wiki setting in Appendix F.
Answer to Reviewer 5tDf [p3]
Question 3
State only encoder training on Full-Wiki
Our original fixed-pool experiments followed Q-RAG’s default configuration, which trains both encoders. We adopted state-only training for computational reasons in Full-Wiki. Updating the action encoder in Full-Wiki would require repeatedly re-encoding 21 million passages and refreshing the vector index. Asynchronous refreshes would additionally require coordinating index updates and handling stale passage embeddings. We therefore train only the state encoder and reuse the fixed corpus index. We agree that Full-Wiki serves as a practical demonstration of reward-driven retrieval adaptation in a different setting.
To isolate the effect of freezing the action encoder, we additionally ran a matched control comparing state-only training with training both encoders. The control uses the HotpotQA fixed-pool setting with Qwen3-4B as the Answerer. For LLM-as-a-Judge, Gemma4-E4B-it provides the training reward. All scores are percentages.
| Training reward | Trainable encoders | Retrieval EM | Retrieval F1 | Answer EM | Answer F1 | Qwen3-4B judge | Qwen3-32B judge |
|---|---|---|---|---|---|---|---|
| LLM-as-a-Judge | State + action | 77.4 | 87.5 | 58.6 | 73.5 | 75.8 | 82.3 |
| LLM-as-a-Judge | State only | 77.3 | 87.4 | 58.4 | 73.2 | 75.6 | 82.1 |
| Support Facts | State + action | 89.1 | 93.7 | 59.4 | 74.3 | 76.7 | 83.3 |
| Support Facts | State only | 88.4 | 93.3 | 59.0 | 74.0 | 76.5 | 83.2 |
Freezing the action encoder decreases Answer F1 by 0.3 percentage points for both rewards. Answer EM decreases by 0.2 points for LLM-as-a-Judge and 0.4 points for Support Facts. Across all six metrics, the largest change is 0.7 points, so every difference is below one percentage point. Support Facts retains its 0.8-point advantage in Answer F1 in both training modes. The additional control shows that freezing the action encoder has only a small effect in the tested fixed-pool configurations.
Answer to Reviewer 5tDf [p1]
We thank the reviewer for their careful reading of our manuscript and for their thoughtful comments and questions. We address these points below.
Weakness #1 and Question #1
Stability of the reward ranking
We tested whether the ranking of training rewards changes with the retriever’s training seed or with the model used to generate and judge answers. We trained Q-RAG on HotpotQA with two seeds for each of six rewards, using Qwen3-4B during training. For each checkpoint, Qwen3-4B and Gemma4-E4B-it each generated and judged answers from the same retrieved passages on the test runs. Values are means ± sample standard deviation across the four seed-to-model combinations.
Ranking the six rewards separately for each of the three metrics in each of the four configurations gave 12 rankings. Nine reproduced the full row order shown above. The other three each differed by one adjacent swap, all under Gemma. The same two rewards therefore held the top two places in all 12 rankings. LLM-as-a-Judge ranked first in 10, including all four Answer EM rankings. Group IG performed better than IG on all 12 rankings.
Switching from Qwen to Gemma while holding the metric fixed gave a mean Spearman rank correlation of 0.971 across the matched comparisons (range 0.943–1.000). Switching the training seed while holding the evaluation model and metric fixed gave the same mean and range. Additionally we measured agreement with Kendall’s coefficient of concordance[1], which equals 1 for identical rankings. Across the 12 rankings, W = 0.975.
[1] Legendre, Pierre. "Species associations: the Kendall coefficient of concordance revisited." Journal of agricultural, biological, and environmental statistics 10.2 (2005): 226-245.
Cross-model validation and training
The previous experiment varied the model family only at evaluation. Here we also varied which model generated training answers and which model judged them, retraining both the state and action encoders in each configuration. For each checkpoint, Qwen3-4B and Gemma4-E4B-it answered from the same retrieved passages. For cross-model training, we focused on the three best performing rewards: LLM-as-a-Judge, Group IG, Exact Match. Each cell reports Answer EM / Answer F1 (%). Bold scores mark the best result for each evaluation answerer.
With either evaluation answerer, the highest EM and F1 came from training with that answerer and a judge from the other family. All three Judge configurations outscored Group IG and Exact Match on F1 with both answerers. Qwen outscored Gemma at evaluation by ~3. points in Answer EM and Answer F1. With Gemma as the training answerer, switching the training judge from Gemma to Qwen increased F1 by only 0.1 point when Qwen answered and 0.2 when Gemma answered. LLM-as-a-Judge retained its lead on HotpotQA under cross-family scoring.
We plan to evaluate and add all remaining rewards, and extend these experiments to 2WikiMultiHopQA in the final version of the paper.