- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BeneRAG, a selective retrieval method for black-box language models. It predicts whether using RAG will improve the current answer. The prediction is based on the question, several no-retrieval answers, and verbalized confidence. The benefit labels are created offline by comparing answers with and without retrieval. Experiments are conducted on pooled SQuAD, HotpotQA, and AdversarialQA data.
Critical strengths
-
The paper studies a practical problem. An incorrect answer does not always mean that retrieval will help.
-
The main controlled comparison is well designed. Q-C, Q-B, F-C, and F-B use the same model architecture and training procedure. This comparison clearly shows that benefit supervision works better than correctness supervision under BeneRAG.
Critical weaknesses
-
It is unclear what the estimator learns. The estimator never sees the retrieved documents. A question-only model already achieves 0.7287 AUROC, close to 0.7498 for the full model. It may mainly learn which questions are difficult or usually benefit from BM25, rather than whether retrieval will help the current answer.
-
Generalization is limited. BeneRAG improves F1 on only two of five unseen generators and performs worse than Always-RAG on the other three. It is also not tested on any unseen dataset. Since BeneRAG is designed for black-box model adaptation, cross-model and cross-task generalization should be a central requirement.
-
The baseline comparison is insufficient. Most comparisons are variants of the same DeBERTa classifier. Self-RAG uses a different generator, while SeaKR and Rowen are only protocol adaptations. These results show that benefit supervision is better than correctness supervision, but they do not show that BeneRAG outperforms existing selective-retrieval methods.
-
The retrieval setting is narrow. The main experiments use BM25 and query-specific pools that contain supporting evidence and a fixed 1:5 distractor ratio. Stronger retrievers are not tested. When distractors are removed, the frozen BeneRAG policy performs much worse than Always-RAG.
Overall, BeneRAG is validated under large-scale supervision, fixed datasets, one main training generator, and a specific BM25 retrieval protocol. Since the method is positioned as a solution for adapting black-box models, stronger generalization across models, tasks, corpora, and retrievers is necessary.
Questions for authors
-
The current generalization experiments mainly change the generator, while the F1 gains are inconsistent. Can the authors evaluate dataset transfer, for example by training on the current datasets and testing on unseen benchmarks such as TriviaQA or Natural Questions?
-
SQuAD, HotpotQA, and AdversarialQA have been widely used in prior RAG studies. Can the authors report results from recent RAG systems, even as reference-only numbers, and discuss how BeneRAG compares with them?
-
Have the authors tested dense retrieval using open-source encoders such as BGE? BM25 is a relatively simple retriever and may reduce the quality of the RAG branch. This could make selective retrieval appear more useful than it would be with a stronger retriever.
5 comments
Response to Reviewer SWbv
W4. Thank you for this important point; we accept it. Without distractors, or with a strong retriever, the gain over Always RAG disappears. Your comment helped us see why this happens and what still holds:
- The labels should change. We learn the benefit of the deployed RAG branch, so when the retriever changes the labels should change too. Retraining on BGE labels restores ranking (AUROC 0.706 → 0.745).
- The core claim still holds with BGE. At equal budget, benefit supervision beats correctness supervision (Q3).
- Less room to skip. With BGE, questions where retrieval helps (ΔF1 > 0.05) rise from 34% to 52%, and those where it hurts (ΔF1 < −0.05) fall from 15% to 8%. A selective policy simply has less room to gain by skipping.
Selective retrieval therefore matters most where retrieval is weak, noisy or costly. We will state this scope clearly in the Limitations.
Q1. Thank you for asking about unseen datasets. We applied the frozen router (τ = 0.21, no retraining or relabeling) to the full validation sets of three datasets it never saw:
| Dataset | Questions | Always LLM | Always RAG | BeneRAG | Benefit AUROC |
|---|---|---|---|---|---|
| TriviaQA | 7,993 | 0.574 | 0.464 | 0.584 | 0.745 |
| NQ | 3,610 | 0.259 | 0.300 | 0.310 | 0.553 |
| MuSiQue (multi-hop) | 2,417 | 0.118 | 0.146 | 0.160 | 0.671 |
All six differences (BeneRAG vs each fixed policy on each dataset) have 95% CIs excluding 0. This includes TriviaQA, where retrieval usually hurts and the router learns to skip it. Ranking is weak on NQ, which we will note as a limitation. These rows will go into the main dataset table.
Q2. Thank you for this suggestion. Related Work will add each paper's reported numbers together with its conditions:
| System (generator) | Dataset | Reported |
|---|---|---|
| Adaptive-RAG (FLAN-T5-XL) | HotpotQA | F1 53.82 |
| SeaKR (LLaMA-2-chat-7B) | HotpotQA | F1 39.7 |
| SCAAR (GPT-4o) | HotpotQA | F1 0.679 |
| LLM-independent RAG (LLaMA-3.1-8B) | NQ | In-Accuracy 50.0 |
| BeneRAG, ours (Qwen2.5-7B) | HotpotQA | F1 45.2 |
These numbers are not directly comparable. The generators, corpora, evaluation subsets (about 300–500 questions) and metrics all differ, and none of these papers reports AdversarialQA. For that reason our direct comparisons use the common protocol in W3.
Q3. Thank you for this valuable suggestion; it became one of our most informative experiments. We used bge-base-en-v1.5 as the retriever. It also re-picked each question's distractors (documents that BGE itself finds confusable) and retrieved the top 3. Our core claim holds: at the same budget, benefit supervision beats correctness supervision. The table shows F1, with the gap to F-C in parentheses:
| Budget | F-C (correctness) | F-B, BM25 labels | F-B, BGE labels |
|---|---|---|---|
| 30% | 0.371 | 0.403 (+0.031) | 0.413 (+0.042) |
| 50% | 0.461 | 0.484 (+0.023) | 0.499 (+0.038) |
| 70% | 0.545 | 0.547 (+0.002, n.s.) | 0.558 (+0.013) |
All other gaps have 95% CIs excluding 0 (one seed). F-C needs no retraining: its target does not involve retrieval, and we verified that its inputs and labels are identical row by row under BM25 and BGE.
What BGE removes is the gain over Always RAG. The RAG branch gets much stronger: the answer appears in the context for 0.68 of questions instead of 0.41, and Always RAG F1 rises from 0.40 to 0.59. The difference to Always RAG is −0.079 for the frozen router and −0.005 for the retrained router (still significant).
Retrieval stays within per-question candidate pools rather than the full corpus. We will state this in the Limitations.
Once again, we thank the reviewer for these insightful questions. They have made our claims narrower but much better supported, and we would be glad to clarify anything further.
Update on Qwen3.5-9B: conclusions unchanged
Update: the answers above hold on Qwen3.5-9B. Reviewer yMwK's question about the generator backbone led us to move the whole paper to a newer generator, Qwen3.5-9B, and the paper is stronger for it: we regenerated all sampled answers, confidences and labels, retrained the routers with 5 seeds, and refined BeneRAG's loss to weight each question by max(|Δ|, 0.05), the F1 that a wrong decision costs (Δ = F1 with retrieval − F1 without). We therefore update the numbers from our answers to you below. The contribution is unchanged: benefit supervision allocates retrieval better than correctness supervision at the same budget. In the two main comparisons, fixed before a single test evaluation of BeneRAG, BeneRAG wins in all 5 seeds: +0.0152 [+0.0100, +0.0205] over Always RAG and +0.0406 [+0.0355, +0.0457] over the correctness router F-C at a 50% retrieval budget. Seed 42 unless noted; brackets are paired question-bootstrap 95% CIs.
W1. Same conclusion: the router also uses the answers.
| Check | Earlier answer | Qwen3.5 |
|---|---|---|
| Benefit AUROC: original / answers swapped / question swapped | 0.750 / 0.707 / 0.557 | 0.719 / 0.658 / 0.578 |
| F1 gain over a question-only router, 30/50/70% | +0.009 to +0.010 | +0.009/+0.014/+0.015 (5 seeds) |
| Same, router trained with the same weighted loss | — | +0.012/+0.014/+0.012 (5 seeds) |
Every per-seed CI is above 0.
W2. Same picture, with more significant results. Frozen BeneRAG on six generators:
| Generator | − Always RAG, F1 | − Always RAG, | − F-C, F1 at 30/50/70% |
|---|---|---|---|
| Qwen2.5-7B | +0.0128 | +0.0567 | +0.0476/+0.0406/+0.0218 |
| GPT-4.1 | +0.0041 (5 seeds: +0.0085) | +0.0652 | +0.0204/+0.0164/+0.0138 |
| Claude Sonnet 4.5 | +0.0102 | +0.0554 | +0.0226/−0.0001/−0.0124 |
| Llama-3.1-8B | −0.0199 | +0.0189 | +0.0460/+0.0340/+0.0169 |
| Phi-3.5-mini | −0.0110 | +0.0311 | +0.0533/+0.0412/+0.0215 |
| EXAONE-3.5-2.4B | −0.0213 | +0.0148 | +0.0698/+0.0523/+0.0300 |
= F1 − 0.1 × retrieval rate. All CIs exclude 0 except GPT-4.1's seed-42 F1 gain (over 5 seeds it is positive in every seed, p = 0.007) and Claude − F-C at 50% (n.s.); Claude at 70% is below F-C, as before.
W3. Same:
| Method | F1 | Retrieval rate | BeneRAG − method |
|---|---|---|---|
| Adaptive-RAG | 0.429 | 94% | +0.012 |
| SKR | 0.419 | 77% | +0.021 |
| BeneRAG | 0.440 | 59% | — |
All three paired CIs exclude 0.
W4. Same, and stronger on the core claim (benefit beats correctness at equal budget) across retrieval settings:
| Setting (retrained) | BeneRAG − F-C, F1 at 30/50/70% | BeneRAG − Always RAG, |
|---|---|---|
| BGE retriever | +0.043/+0.036/+0.015 | +0.0073 |
| No distractors | +0.035/+0.032/+0.012 | +0.0126 |
| Shared SQuAD corpus | +0.047/+0.040/+0.024 (5 seeds) | +0.0202 |
All CIs exclude 0. Retraining on BGE labels restores ranking (AUROC 0.671 → 0.714). As before, the F1 gain over Always RAG disappears without distractors or with BGE.
Q1. Same: all six differences remain significant; ranking is still weakest on NQ.
| Dataset | Always LLM | Always RAG | BeneRAG | Benefit AUROC |
|---|---|---|---|---|
| TriviaQA | 0.626 | 0.589 | 0.666 | 0.758 |
| NQ | 0.275 | 0.330 | 0.351 | 0.572 |
| MuSiQue | 0.125 | 0.157 | 0.187 | 0.673 |
Q2. BeneRAG's HotpotQA F1 is now 49.1 (5 seeds); the other reported numbers are unchanged.
Q3. Same, with bge-base-en-v1.5 (F1; gap to F-C in parentheses):
| Budget | F-C | BeneRAG, BM25 labels | BeneRAG, BGE labels |
|---|---|---|---|
| 30% | 0.3826 | 0.4183 (+0.0358) | 0.4253 (+0.0427) |
| 50% | 0.4815 | 0.5055 (+0.0240) | 0.5176 (+0.0362) |
| 70% | 0.5708 | 0.5696 (−0.0013, n.s.) | 0.5855 (+0.0147) |
All other gaps are significant. BeneRAG − Always RAG is −0.088 frozen and −0.005 retrained (still significant), as before.
Response to Reviewer SWbv
We sincerely thank the reviewer for the careful and insightful review. Your questions about what the estimator really learns and how far it generalizes were the most searching ones we received. Answering them changed how we understand our own method. We are grateful for this, and we respond to every point in full below.
Terminology:
All intervals are paired question-level bootstrap 95% CIs.
W1. Thank you for this sharp question. Your suspicion is partly right: much of the decision comes from the question itself, although the router also uses the answers. To test this directly, we fed the trained router mismatched inputs:
Replacing the question destroys most of the ranking. Replacing the answers lowers it noticeably but less. At matched budgets, F-B's F1 is 0.009–0.010 higher than the question-only router.
Your comment also led us to a stricter test of whether the router reads individual answers:
So, as you suspected, BeneRAG's advantage comes mainly from choosing which questions to retrieve for, rather than from reading one particular answer. We will describe the method in these terms and put both experiments in the appendix.
W2. Thank you for this fair criticism; we accept it. Without retraining, the three open models are 0.009–0.023 F1 below Always RAG, although ranking survives (AUROC 0.71–0.73).
We examined why. Even a perfect router would gain only about half as much for these models as for the API models (0.054–0.061 vs 0.103–0.104). This is consistent with the losses, but we have not proven it to be their cause.
What does transfer is the choice of which questions to retrieve for, compared at the same budget:
The pattern holds for four of the five generators. Claude is the exception at the larger budgets.
Two further adaptation tests:
Unseen datasets are covered in Q1 (Part 2). Following your comment, we will limit the generalization claim to ranking and matched-budget transfer.
W3. We are grateful for this suggestion. Under a common protocol, every method uses the same 10,000 questions and answer branches, with its threshold tuned on validation. We use Adaptive-RAG as a two-branch classifier because our pipeline has a single retrieval step.
BeneRAG retrieves less and reaches higher F1. Both differences remain significant after Holm correction (one training seed per method). These methods will go into the main comparison figure and the appendix table.