acceptodds
Under review as a conference paper at ICLR 2027

BeneRAG: Benefit Supervision for Black-Box Selective Retrieval

Abstract

Answer correctness alone does not determine whether retrieval will improve a language model's response. Retrieval may leave an incorrect answer unimproved when it provides insufficient evidence. We study black-box selective retrieval through protocol-conditional context benefit, the improvement obtained by executing the deployed retrieval-augmented generation (RAG) branch. BeneRAG learns from paired branch outcomes offline and predicts whether retrieval will improve the current answer using only the question, sampled no-retrieval answers, and verbalized confidence. With observations, architecture, and training held fixed, benefit supervision yields higher F1 than correctness supervision at every tested budget in exact-budget ranking evaluations, gaining F1 points at a % invocation rate. A question-only benefit estimator also outperforms the full-observation correctness estimator at these budgets. Across three QA datasets with controlled evidence availability, the policy improves answer quality over Always RAG while invoking retrieval less often. Automatic semantic evaluation with GPT-4.1 also favors BeneRAG over Always RAG. Benefit ranking remains useful under corpus and generator changes, but quality gains depend on the protocol and operating point. Acquiring the observations also incurs additional execution cost. These results support supervising the incremental benefit of the available RAG branch when allocating a limited retrieval budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Retrieval-Augmented Generation, Selective Retrieval, Benefit Supervision, Black-Box Language Models, Retrieval Routing, Question Answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SWbvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BeneRAG, a selective retrieval method for black-box language models. It predicts whether using RAG will improve the current answer. The prediction is based on the question, several no-retrieval answers, and verbalized confidence. The benefit labels are created offline by comparing answers with and without retrieval. Experiments are conducted on pooled SQuAD, HotpotQA, and AdversarialQA data.

Critical strengths
  1. The paper studies a practical problem. An incorrect answer does not always mean that retrieval will help.

  2. The main controlled comparison is well designed. Q-C, Q-B, F-C, and F-B use the same model architecture and training procedure. This comparison clearly shows that benefit supervision works better than correctness supervision under BeneRAG.

Critical weaknesses
  1. It is unclear what the estimator learns. The estimator never sees the retrieved documents. A question-only model already achieves 0.7287 AUROC, close to 0.7498 for the full model. It may mainly learn which questions are difficult or usually benefit from BM25, rather than whether retrieval will help the current answer.

  2. Generalization is limited. BeneRAG improves F1 on only two of five unseen generators and performs worse than Always-RAG on the other three. It is also not tested on any unseen dataset. Since BeneRAG is designed for black-box model adaptation, cross-model and cross-task generalization should be a central requirement.

  3. The baseline comparison is insufficient. Most comparisons are variants of the same DeBERTa classifier. Self-RAG uses a different generator, while SeaKR and Rowen are only protocol adaptations. These results show that benefit supervision is better than correctness supervision, but they do not show that BeneRAG outperforms existing selective-retrieval methods.

  4. The retrieval setting is narrow. The main experiments use BM25 and query-specific pools that contain supporting evidence and a fixed 1:5 distractor ratio. Stronger retrievers are not tested. When distractors are removed, the frozen BeneRAG policy performs much worse than Always-RAG.

Overall, BeneRAG is validated under large-scale supervision, fixed datasets, one main training generator, and a specific BM25 retrieval protocol. Since the method is positioned as a solution for adapting black-box models, stronger generalization across models, tasks, corpora, and retrievers is necessary.

Questions for authors
  1. The current generalization experiments mainly change the generator, while the F1 gains are inconsistent. Can the authors evaluate dataset transfer, for example by training on the current datasets and testing on unseen benchmarks such as TriviaQA or Natural Questions?

  2. SQuAD, HotpotQA, and AdversarialQA have been widely used in prior RAG studies. Can the authors report results from recent RAG systems, even as reference-only numbers, and discuss how BeneRAG compares with them?

  3. Have the authors tested dense retrieval using open-source encoders such as BGE? BM25 is a relatively simple retriever and may reduce the quality of the RAG branch. This could make selective retrieval appear more useful than it would be with a stronger retriever.

5 comments
AuthorsOpenReview ↗

Response to Reviewer SWbv

We sincerely thank the reviewer for the careful and insightful review. Your questions about what the estimator really learns and how far it generalizes were the most searching ones we received. Answering them changed how we understand our own method. We are grateful for this, and we respond to every point in full below.

Terminology:

  • BeneRAG (F-B): a router that reads the question, three sampled no-retrieval answers and the model's verbalized confidence, and predicts whether retrieval will improve the answer (benefit).
  • F-C: the same router trained instead to predict whether the no-retrieval answer is wrong (correctness).
  • Q-B: a benefit router that sees only the question.

All intervals are paired question-level bootstrap 95% CIs.

W1. Thank you for this sharp question. Your suspicion is partly right: much of the decision comes from the question itself, although the router also uses the answers. To test this directly, we fed the trained router mismatched inputs:

Router inputBenefit AUROC
Original question and its answers0.750
Answers and confidence taken from another question0.707
Question taken from another question0.557

Replacing the question destroys most of the ranking. Replacing the answers lowers it noticeably but less. At matched budgets, F-B's F1 is 0.009–0.010 higher than the question-only router.

Your comment also led us to a stricter test of whether the router reads individual answers:

  • We regenerated the no-retrieval answers 5 times for each of 7,000 questions, keeping the RAG answer fixed.
  • For the 323 questions whose benefit label changes across repetitions, we asked which repetition would benefit from retrieval.
  • BeneRAG was no better than the common correctness router (within-question AUROC 0.579 vs 0.616; the difference is not significant).

So, as you suspected, BeneRAG's advantage comes mainly from choosing which questions to retrieve for, rather than from reading one particular answer. We will describe the method in these terms and put both experiments in the appendix.

W2. Thank you for this fair criticism; we accept it. Without retraining, the three open models are 0.009–0.023 F1 below Always RAG, although ranking survives (AUROC 0.71–0.73).

We examined why. Even a perfect router would gain only about half as much for these models as for the API models (0.054–0.061 vs 0.103–0.104). This is consistent with the losses, but we have not proven it to be their cause.

What does transfer is the choice of which questions to retrieve for, compared at the same budget:

GeneratorF-B − F-C (budget 30/50/70%)
GPT-4.1+0.017/+0.013/+0.007
Claude Sonnet 4.5+0.012/−0.012/−0.022
Llama-3.1-8B+0.045/+0.032/+0.011
Phi-3.5-mini+0.052/+0.039/+0.020
EXAONE-3.5-2.4B+0.069/+0.054/+0.029

The pattern holds for four of the five generators. Claude is the exception at the larger budgets.

Two further adaptation tests:

  • Threshold only. Re-fitting only the threshold on 2,000 target labels, with the goal of preserving F1, narrows the F1 gap to 0.0006–0.0018. The cost is a higher retrieval rate (73–91%).
  • Full retraining. Retraining everything on a second generator (Qwen3.5-9B, 5 seeds) keeps F-B above F-C at all budgets.

Unseen datasets are covered in Q1 (Part 2). Following your comment, we will limit the generalization claim to ranking and matched-budget transfer.

W3. We are grateful for this suggestion. Under a common protocol, every method uses the same 10,000 questions and answer branches, with its threshold tuned on validation. We use Adaptive-RAG as a two-branch classifier because our pipeline has a single retrieval step.

MethodF1Retrieval rateBeneRAG − method, F1 [95% CI]
Adaptive-RAG (two-branch)0.40393%+0.012 [+0.007, +0.018]
SKR0.40075%+0.015 [+0.010, +0.021]
BeneRAG0.41558%—

BeneRAG retrieves less and reaches higher F1. Both differences remain significant after Holm correction (one training seed per method). These methods will go into the main comparison figure and the appendix table.

AuthorsOpenReview ↗

Response to Reviewer SWbv

W4. Thank you for this important point; we accept it. Without distractors, or with a strong retriever, the gain over Always RAG disappears. Your comment helped us see why this happens and what still holds:

  • The labels should change. We learn the benefit of the deployed RAG branch, so when the retriever changes the labels should change too. Retraining on BGE labels restores ranking (AUROC 0.706 → 0.745).
  • The core claim still holds with BGE. At equal budget, benefit supervision beats correctness supervision (Q3).
  • Less room to skip. With BGE, questions where retrieval helps (ΔF1 > 0.05) rise from 34% to 52%, and those where it hurts (ΔF1 < −0.05) fall from 15% to 8%. A selective policy simply has less room to gain by skipping.

Selective retrieval therefore matters most where retrieval is weak, noisy or costly. We will state this scope clearly in the Limitations.

Q1. Thank you for asking about unseen datasets. We applied the frozen router (τ = 0.21, no retraining or relabeling) to the full validation sets of three datasets it never saw:

DatasetQuestionsAlways LLMAlways RAGBeneRAGBenefit AUROC
TriviaQA7,9930.5740.4640.5840.745
NQ3,6100.2590.3000.3100.553
MuSiQue (multi-hop)2,4170.1180.1460.1600.671

All six differences (BeneRAG vs each fixed policy on each dataset) have 95% CIs excluding 0. This includes TriviaQA, where retrieval usually hurts and the router learns to skip it. Ranking is weak on NQ, which we will note as a limitation. These rows will go into the main dataset table.

Q2. Thank you for this suggestion. Related Work will add each paper's reported numbers together with its conditions:

System (generator)DatasetReported
Adaptive-RAG (FLAN-T5-XL)HotpotQAF1 53.82
SeaKR (LLaMA-2-chat-7B)HotpotQAF1 39.7
SCAAR (GPT-4o)HotpotQAF1 0.679
LLM-independent RAG (LLaMA-3.1-8B)NQIn-Accuracy 50.0
BeneRAG, ours (Qwen2.5-7B)HotpotQAF1 45.2

These numbers are not directly comparable. The generators, corpora, evaluation subsets (about 300–500 questions) and metrics all differ, and none of these papers reports AdversarialQA. For that reason our direct comparisons use the common protocol in W3.

Q3. Thank you for this valuable suggestion; it became one of our most informative experiments. We used bge-base-en-v1.5 as the retriever. It also re-picked each question's distractors (documents that BGE itself finds confusable) and retrieved the top 3. Our core claim holds: at the same budget, benefit supervision beats correctness supervision. The table shows F1, with the gap to F-C in parentheses:

BudgetF-C (correctness)F-B, BM25 labelsF-B, BGE labels
30%0.3710.403 (+0.031)0.413 (+0.042)
50%0.4610.484 (+0.023)0.499 (+0.038)
70%0.5450.547 (+0.002, n.s.)0.558 (+0.013)

All other gaps have 95% CIs excluding 0 (one seed). F-C needs no retraining: its target does not involve retrieval, and we verified that its inputs and labels are identical row by row under BM25 and BGE.

What BGE removes is the gain over Always RAG. The RAG branch gets much stronger: the answer appears in the context for 0.68 of questions instead of 0.41, and Always RAG F1 rises from 0.40 to 0.59. The difference to Always RAG is −0.079 for the frozen router and −0.005 for the retrained router (still significant).

Retrieval stays within per-question candidate pools rather than the full corpus. We will state this in the Limitations.

Once again, we thank the reviewer for these insightful questions. They have made our claims narrower but much better supported, and we would be glad to clarify anything further.

AuthorsOpenReview ↗

Update on Qwen3.5-9B: conclusions unchanged

Update: the answers above hold on Qwen3.5-9B. Reviewer yMwK's question about the generator backbone led us to move the whole paper to a newer generator, Qwen3.5-9B, and the paper is stronger for it: we regenerated all sampled answers, confidences and labels, retrained the routers with 5 seeds, and refined BeneRAG's loss to weight each question by max(|Δ|, 0.05), the F1 that a wrong decision costs (Δ = F1 with retrieval − F1 without). We therefore update the numbers from our answers to you below. The contribution is unchanged: benefit supervision allocates retrieval better than correctness supervision at the same budget. In the two main comparisons, fixed before a single test evaluation of BeneRAG, BeneRAG wins in all 5 seeds: +0.0152 [+0.0100, +0.0205] over Always RAG and +0.0406 [+0.0355, +0.0457] over the correctness router F-C at a 50% retrieval budget. Seed 42 unless noted; brackets are paired question-bootstrap 95% CIs.

W1. Same conclusion: the router also uses the answers.

CheckEarlier answerQwen3.5
Benefit AUROC: original / answers swapped / question swapped0.750 / 0.707 / 0.5570.719 / 0.658 / 0.578
F1 gain over a question-only router, 30/50/70%+0.009 to +0.010+0.009/+0.014/+0.015 (5 seeds)
Same, router trained with the same weighted loss—+0.012/+0.014/+0.012 (5 seeds)

Every per-seed CI is above 0.

W2. Same picture, with more significant results. Frozen BeneRAG on six generators:

Generator− Always RAG, F1− Always RAG, UcallU_{call}− F-C, F1 at 30/50/70%
Qwen2.5-7B+0.0128+0.0567+0.0476/+0.0406/+0.0218
GPT-4.1+0.0041 (5 seeds: +0.0085)+0.0652+0.0204/+0.0164/+0.0138
Claude Sonnet 4.5+0.0102+0.0554+0.0226/−0.0001/−0.0124
Llama-3.1-8B−0.0199+0.0189+0.0460/+0.0340/+0.0169
Phi-3.5-mini−0.0110+0.0311+0.0533/+0.0412/+0.0215
EXAONE-3.5-2.4B−0.0213+0.0148+0.0698/+0.0523/+0.0300

UcallU_{call} = F1 − 0.1 × retrieval rate. All CIs exclude 0 except GPT-4.1's seed-42 F1 gain (over 5 seeds it is positive in every seed, p = 0.007) and Claude − F-C at 50% (n.s.); Claude at 70% is below F-C, as before.

W3. Same:

MethodF1Retrieval rateBeneRAG − method
Adaptive-RAG0.42994%+0.012
SKR0.41977%+0.021
BeneRAG0.44059%—

All three paired CIs exclude 0.

W4. Same, and stronger on the core claim (benefit beats correctness at equal budget) across retrieval settings:

Setting (retrained)BeneRAG − F-C, F1 at 30/50/70%BeneRAG − Always RAG, UcallU_{call}
BGE retriever+0.043/+0.036/+0.015+0.0073
No distractors+0.035/+0.032/+0.012+0.0126
Shared SQuAD corpus+0.047/+0.040/+0.024 (5 seeds)+0.0202

All CIs exclude 0. Retraining on BGE labels restores ranking (AUROC 0.671 → 0.714). As before, the F1 gain over Always RAG disappears without distractors or with BGE.

Q1. Same: all six differences remain significant; ranking is still weakest on NQ.

DatasetAlways LLMAlways RAGBeneRAGBenefit AUROC
TriviaQA0.6260.5890.6660.758
NQ0.2750.3300.3510.572
MuSiQue0.1250.1570.1870.673

Q2. BeneRAG's HotpotQA F1 is now 49.1 (5 seeds); the other reported numbers are unchanged.

Q3. Same, with bge-base-en-v1.5 (F1; gap to F-C in parentheses):

BudgetF-CBeneRAG, BM25 labelsBeneRAG, BGE labels
30%0.38260.4183 (+0.0358)0.4253 (+0.0427)
50%0.48150.5055 (+0.0240)0.5176 (+0.0362)
70%0.57080.5696 (−0.0013, n.s.)0.5855 (+0.0147)

All other gaps are significant. BeneRAG − Always RAG is −0.088 frozen and −0.005 retrained (still significant), as before.

Reviewer_yMwKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper argues that answer correctness alone does not determine whether retrieval helps in retrieval-augmented generation (RAG) in Question Answering (QA). To address this limitation in RAG, the paper studies black-box selective retrieval through protocol-conditional context benefit and proposes BeneRAG, which learns from paired branch outcomes offline and predicts whether retrieval will improve the answer. Specifically, BeneRAG first collects training signals by comparing two branches: one branch with no retrieved context (closed-book), and one normal RAG branch with additional context. It then trains a simple binary DeBERTa model to predict whether retrieval will help answer the question or not, given the question, sampled answers and verbalised self-confidence. Controlled experiments on three QA datasets show that BeneRAG achieves higher overall F1 scores while reducing retrieval frequency.

Overall, this novelty contribution of the paper is limited, and the proposed BeneRAG has obvious shortcomings and lacks extensive validation, limiting its practical usage in reality.

Critical strengths
  1. The motivation of the paper is interesting, and the problem this paper aims to address is indeed a limitation of traditional RAG. It first collects training signals by comparing two branches: one branch with no retrieved context (closed-book), and one normal RAG branch with additional context. It then trains a simple binary DeBERTa model to predict whether retrieval will help answer the question or not, given the question, sampled answers and verbalised self-confidence.
  2. The experiments seem to be extensive, yet the methodology might be problematic (see below).
Critical weaknesses
  1. The proposed method, BeneRAG, might not be practical and has several potential issues.
    • It essentially requires collecting offline benefit signals on a given labelled dataset by running inference twice: once with retrieval and once without retrieval. In real-world scenarios, collecting high-quality labelled datasets is itself challenging. Therefore, obtaining offline benefit supervision requires additional effort to label the data.
    • While the paper analyses the transferability of the generator (LLM), it does not investigate the transferability of the trained Benefit Estimator. Does the Benefit Estimator trained on the three QA datasets generalise to other datasets? Would a new Benefit Estimator need to be trained? If so, new benefit signals would need to be collected again on a new labelled dataset.
    • The performance of the trained binary Benefit Estimator and the verbalised self-assessment lacks essential evaluation and analysis. Previous works have shown that LLMs' judgements of their own answers are not reliable. Yet, the paper uses self-confidence as one of the training signals without sufficient justification. In addition, the benefit signals can be regarded as pseudo-labels for the questions. However, the correctness of these benefit signals (pseudo-labels) is not sufficiently studied. Can we trust them?
    • The backbone generator (LLM) used in the experiments is relatively old (Qwen-2.5). The experimental findings might no longer hold when using more recent models, such as Qwen-3/3.5. Furthermore, the performance gains reported in the experiments are marginal compared to the baselines, making the effectiveness of the proposed method questionable.
  2. The established problem and experimental settings are limited to simple, short-form QA, which might not be practical in real-world scenarios. In reality, many questions across different domains require long-form answers, and complex questions may also require iterative, advanced reasoning. However, the scope of the framed problem is somewhat narrow, and the datasets used in the experiments are also relatively outdated (released in 2016, 2018, and 2020). Modern LLMs might already perform well using only their internal knowledge without retrieval.
Questions for authors
  1. why not using more advanced models, such as Qwen 3 or 3.5 series as the base model?
  2. The threshold ϵ\epsilon is selected on the validation split to maximize UcallU_{call}, does this value bound to each dataset or for all three datasets, they are using the same value?
7 comments
AuthorsOpenReview ↗

Response to Reviewer yMwK

We sincerely thank the reviewer for the careful and constructive review, and we apologize for our late response. Following your comment on the backbone, we moved the whole paper to a newer generator, Qwen3.5-9B, and re-ran every experiment, which took considerable time. The overall story of the paper is unchanged. Analyzing the labels of the new backbone led to one refinement of the training loss ((b) below), and our contribution, that benefit supervision allocates retrieval better than correctness supervision at the same budget, is unchanged. Your comments on labeling cost, transfer, reliability, the backbone and scope pointed to exactly the places where our paper needed stronger evidence, and answering them has made it stronger. We answer each point below, in the order of the review.

Summary of changes.

  • (a) New backbone. The whole paper now uses Qwen3.5-9B as the generator. We regenerated all sampled answers, confidences and labels, and retrained BeneRAG, F-C and Q-B with 5 seeds (Q1); results marked seed 42 use one training seed.
  • (b) A loss refinement motivated by the new labels. On Qwen3.5, most questions where retrieval does not help are ties: 51.6% of training and validation questions have |Δ| ≤ 0.05, and only 14.2% are harmful. A selective policy beats Always RAG by skipping the harmful cases, and a wrong decision on a question costs exactly |Δ| in F1. BeneRAG's loss therefore now weights each question by max(|Δ|, 0.05), normalized to mean 1, so that training counts each mistake the way the evaluation does. Label, inputs, encoder and threshold rule are unchanged.
  • (c) Validation-only selection, one test evaluation of BeneRAG. The loss was chosen on validation data only, and two planned comparisons, the threshold rule and the success criteria were fixed before a single test evaluation of BeneRAG. BeneRAG wins both in all 5 seeds: BeneRAG − Always RAG = +0.0152 [+0.0100, +0.0205], and BeneRAG − F-C at a 50% budget = +0.0406 [+0.0355, +0.0457] (W4). Other test numbers in this answer are descriptive; none of them changed the method.
  • (d) Generalization checks. Because a loss chosen on our three training datasets could overfit them, we applied the trained router, without retraining, to data it never saw. On TriviaQA, NQ and MuSiQue, which were not used in training, it significantly beats Always RAG: +0.0765, +0.0206 and +0.0299 (all CIs above 0; W2, W5).

We describe each change, together with its limitations, under the corresponding point.

Terminology (all routers below are trained with Qwen3.5-9B as the generator):

  • BeneRAG: our final method. It reads the question, three sampled no-retrieval answers and the model's verbalized confidence, and predicts whether retrieval will improve the answer. Label: z = 1[Δ > 0.05], where Δ = F1 with retrieval − F1 without. It is trained with the weighted loss of (b).
  • F-C: the same inputs and encoder, trained to predict whether the no-retrieval answer is wrong (correctness).
  • Q-B: a benefit router that sees only the question. Weighted Q-B is Q-B trained with BeneRAG's weighted loss (seed 42), for comparisons at equal weighting.
  • Exact budget: each router retrieves for its own top 30/50/70% of questions.
  • UcallU_{call} = F1 − 0.1 × retrieval rate (RR), a utility that charges each retrieval.

Unless noted, results are on the 10,000 test questions, averaged over 5 training seeds (42–46), with paired question-level bootstrap 95% CIs (10,000 resamples).

W1. Thank you for this practical concern. You are right that training requires QA data with gold answers and one run of the generator with and without retrieval. What it does not require is any human annotation of retrieval benefit: the label is computed automatically by comparing the F1 of the two answers against the existing gold answers. This is a one-time, offline cost on QA data that already exists, and the trained router is then used without any further labeling.

Your comment led us to test how far a trained router can be reused without new labels, and how cheap it can be at inference:

  • New datasets. Applied unchanged to TriviaQA, NQ and MuSiQue, which were not used in training, BeneRAG beats Always RAG on each (+0.0765, +0.0206 and +0.0299; all CIs above 0; W2, W5).
  • Fewer samples. With one sampled answer instead of three, the generation cost per question falls from 110 to 79 units (output-token equivalents; Always RAG: 55), and F1 stays above Always RAG (0.4437 vs 0.4283, seed 42), at slightly lower ranking quality (benefit AUROC 0.708 vs 0.719).

In practice, labels are needed once per generator–retriever setting, and their cost is one run of both branches on existing QA data. In the paper, we will state this cost explicitly in the Limitations and put these results in the appendix.

AuthorsOpenReview ↗

Response to Reviewer yMwK

W2. We are grateful for this question; it gave us the clearest evidence on reuse. Whether the estimator generalizes was your concern, and our loss refinement made it all the more important to check, since a loss chosen on our three training datasets could overfit them. We therefore ran dataset-transfer experiments: a leave-one-dataset-out check on validation, and zero-shot transfer to TriviaQA, NQ and MuSiQue, which were not used in training. For the latter, we applied the trained router unchanged (seed 42; threshold τ = 0.57 chosen on the original validation set; no retraining, relabeling or target labels) to the full validation sets of the three datasets:

DatasetQuestionsAlways LLMAlways RAGBeneRAGBeneRAG − Always RAG [95% CI]BeneRAG RRBenefit AUROC
TriviaQA7,9930.6260.5890.666+0.0765 [+0.0697, +0.0834]0.4820.758
NQ3,6100.2750.3300.351+0.0206 [+0.0099, +0.0313]0.5660.572
MuSiQue2,4170.1250.1570.187+0.0299 [+0.0201, +0.0395]0.4940.673
  • On TriviaQA, where the model does better without retrieval (Always LLM 0.626 > Always RAG 0.589), BeneRAG retrieves for 48% of questions and beats both fixed policies (+0.0398 [+0.0319, +0.0479] over Always LLM). On NQ and MuSiQue it also beats both.
  • These are single-seed results. A second, independently trained copy of the router (different training environment) also beats Always RAG on each, with all 95% CIs above 0 (+0.0808, +0.0178, +0.0269).
  • Transfer is not guaranteed when a whole type of data is missing from training. In the leave-one-dataset-out check on validation (seed 42), BeneRAG trained without SQuAD trails Always RAG on SQuAD (−0.0553 [−0.0616, −0.0489]), while held-out HotpotQA (+0.0081) and AdversarialQA (+0.0196) still gain.

In the paper, we will add this table to the dataset-transfer section. So, to answer your question directly: the router often transfers without new labels, but not always, and a new domain may need new labels.

AuthorsOpenReview ↗

Response to Reviewer yMwK

W3. Thank you for questioning the reliability of self-reported confidence, of the estimator and of the labels; we agree that each needs evidence, and your comment led us to the checks below. We do not read verbalized confidence as a probability: it is one observation input, next to the question and the sampled answers.

Verbalized confidence. Share of answers in each stated-confidence group, and the accuracy (F1 ≥ 0.5) of those answers (test):

Stated confidence0–0.50.5–0.80.8–0.90.9–1ECEAUROC vs correctness
Qwen2.5-7B: share / accuracy21% / 0.094% / 0.1252% / 0.2023% / 0.470.470.71
Qwen3.5-9B: share / accuracy10% / 0.083% / 0.053% / 0.0884% / 0.270.640.69

Qwen3.5 is even more overconfident than Qwen2.5: 84% of its answers are rated 0.9 or higher, but only 27% of those are correct, and below 0.9 accuracy stays at 5–8% whatever the stated value. The confidence still carries some signal (AUROC 0.69), mostly by separating the top group from the rest.

Does the router need it? We retrained BeneRAG (seed 42) with the confidence line removed from its input, keeping the question and the samples:

BeneRAG, seed 42With confidenceWithoutDifference [95% CI]
Benefit AUROC0.71890.7136+0.0053 [+0.0024, +0.0082]
F1 at exact 30/50/70% budgets0.3777 / 0.4263 / 0.44930.3767 / 0.4244 / 0.4489+0.0010 / +0.0019 / +0.0004 (all CIs include 0)
F1 (RR) at its own threshold0.4404 (0.594)0.4419 (0.631)−0.0015 [−0.0041, +0.0012]

Confidence is therefore an auxiliary signal for ranking, not a core feature of the method. In this seed-42 ablation it adds a small ranking gain (AUROC +0.0053 [+0.0024, +0.0082]), and removing it changes F1 at equal budgets by at most 0.002 (not significant). We keep it as part of the method fixed before the test evaluation.

The estimator's scores and the labels.

CheckQwen2.5-7B (submitted)Qwen3.5-9B (revision)
Labels rebuilt from other samples (same RAG answer; all 193,216 questions): agreement, κ97.4–99.1%, 0.94–0.9894.6–98.3%, 0.88–0.96
Label identical under all five single samples94.1%88.3%
  • BeneRAG's score is used only to rank questions: the router retrieves for questions whose score exceeds a threshold chosen on validation, or for the top-scored questions under a budget. Within one generator–retriever setting, only the order of the scores enters a decision, so we evaluate the estimator by its ranking quality (benefit AUROC) rather than as a probability.
  • The labels are computed, not annotated: Δ compares the F1 of both answers with the gold answer, so they inherit the limits of token F1. Beyond that, noise comes from the 0.05 margin and from sampling. Moving the margin to 0 or 0.10 changes only 0.29% or 0.59% of the labels. Sampling noise is larger with Qwen3.5 than with Qwen2.5 (κ 0.88–0.96 vs 0.94–0.98). On the test questions whose label is stable, BeneRAG's benefit AUROC is 0.738, versus 0.610 on the rest, so part of the measured ranking error is label noise.

In the paper, we will put these tables in the appendix.

Reviewer_TesUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. BeneRAG is a black-box gate that decides per question whether to retrieve. With the retrieval pipeline fixed, it compares the answer with and without retrieval offline and labels a question "retrieval helps" when the F1 gain exceeds 0.05.
  2. A DeBERTa classifier reads the question, three sampled no-retrieval answers, and a verbalized confidence, and predicts that label. Retrieval is triggered when the probability passes a threshold tuned on validation.
  3. In a controlled 2x2 (label x input) on pooled SQuAD / HotpotQA / AdversarialQA with Qwen2.5-7B, benefit supervision beats correctness supervision at every retrieval budget, and the policy edges out Always RAG (.415 vs .400 F1) while retrieving on 58% of questions.
Critical strengths
  1. The 2x2 control is the right experiment. Same inputs, architecture and training, only the label changes, and the benefit-trained estimator reaches AUROC .750 where the correctness-trained one gets .567 on the same inputs. This is convincing evidence that "will retrieval help" is a better training target than "is the answer wrong".
  2. The method only needs text outputs, so it works with API models.
  3. The appendices are candid: cost accounting, protocol sensitivity (Tables 18, 19), a decision-exchange test, and an error analysis.
Critical weaknesses
  1. The selector learns behavior tied to the question distribution and the retrieval pipeline, and the paper's own numbers show it. The main result uses query-specific pools with 5 injected distractors per gold chunk. Remove them and the frozen policy loses .092 F1 to Always RAG, with the sign flipping between 1:1 and 1:2 (Table 18); retraining recovers +.001 (Table 19). In the shared-corpus setting the gain is +.009 (Table 2). Any change in corpus or retriever means relabeling and retraining, and the abstract does not say the sign depends on the distractor ratio.
  2. The binary label lumps together causes that need different remedies: missing knowledge, poor retrieval, conflicting evidence, reasoning errors. The error analysis does not say which of these drive the gain over correctness supervision, or what happens to the policy when their mix shifts. This is the question I most want answered.
  3. The strong baseline is missing. A router that first detects whether the question needs information the model lacks, then retrieves, is the obvious comparison. SKR [1] builds its labels the same way (answer with vs. without retrieval) and trains a question classifier, and it is not cited. Adaptive-RAG [2] is cited but not run.
  4. Cost. Three draft answers plus a confidence call come to 1.22 GPU-s per query vs .35 for Always RAG (Table 14). The question-only variant is close (AUROC .729 vs .750, Table 9) and the decision-exchange CI includes zero (Table 10), so the expensive part is not shown to matter. The paper's own line that "fewer retrieval calls alone do not establish an efficiency gain" undercuts the budget motivation.
Questions for authors
  1. Which failure types produce the gain? Split the test set by cause using your error analysis (missing knowledge / poor retrieval / conflicting evidence / reasoning error) and report where benefit supervision beats correctness supervision.
  2. Exact-budget F-B vs F-C in the shared-corpus setting, with across-seed std. App. G.4 says the current intervals are question bootstraps, so I cannot tell whether the .009 and .001 gaps are outside seed variance.
  3. Compare against SKR, Adaptive-RAG, and a detect-missing-information-then-retrieve router under your protocol, thresholds tuned on the same validation set.
  4. Is there a realistic retrieval setting (e.g. the Recall@3 of .68 in H.3) where BeneRAG beats Always RAG on both F1 and compute?
4 comments
AuthorsOpenReview ↗

Response to Reviewer TesU

We sincerely thank the reviewer for the careful and thoughtful review. Your questions pushed us to examine exactly where BeneRAG's advantage comes from, and several of the analyses below exist only because of them. We are grateful for this, and we have tried to answer every point in full.

To keep the answers readable, we use the paper's names throughout:

  • F-B (BeneRAG): the router that reads the question, three sampled no-retrieval answers and the model's verbalized confidence, and is trained to predict whether retrieval improves the answer (benefit).
  • F-C: the same inputs, encoder and training recipe, but trained to predict whether the no-retrieval answer is wrong (correctness). This is the natural "retrieve when unsure" alternative.
  • Q-B: a benefit router that sees the question only.
  • Exact budget: retrieve for the top 30/50/70% of questions ranked by the router score, so every router uses the same number of retrieval calls.
  • UcallU_{call} = F1 − 0.1 × retrieval rate (RR).

All intervals are paired question-level bootstrap 95% CIs (10,000 resamples).

W1. Thank you for this important point. We agree that whether BeneRAG beats Always RAG depends on the retrieval environment, and we will state this plainly in the abstract and the Limitations. Your comment made us separate two claims that the original draft blurred:

  1. BeneRAG beats Always RAG. This depends on the environment.
  2. Benefit supervision allocates retrieval better than correctness supervision at the same budget. This is our core claim, and it holds in every environment we tested.
SettingF-B − Always RAG (F1)F-B − F-C, budget 30/50/70%
Canonical (1:5 distractors)+0.015+0.049/+0.037/+0.013
No distractors−0.092 frozen, +0.001 retrained—
Shared corpus−0.017 frozen, +0.009 retrained+0.047/+0.044/+0.031 (5 seeds)
BGE dense retriever−0.079 frozen, −0.005 retrained+0.042/+0.038/+0.013 (1 seed)

"Frozen" means the router trained in the canonical setting is applied unchanged; "retrained" means the benefit labels are rebuilt in the new environment.

When retrieval becomes much stronger (no distractors, or BGE), retrieval helps on most questions, so a policy that sometimes skips retrieval has little left to gain over Always RAG. Even then, at the same budget the benefit router still chooses better which questions deserve retrieval than the correctness router does.

In the BGE row only F-B needs retraining. F-C's target (is the no-retrieval answer wrong?) does not involve retrieval. We verified that its inputs and labels are identical, row by row, under BM25 and BGE, so the canonical and BGE rows compare against the same F-C. We will add this scope statement to the Limitations.

W2. Thank you for asking which failure types drive the gain; we consider this the most important question in the review. It led us to build a full error decomposition, a mix-shift check and a replication on four further generators. The full answer is in Q1 (Part 2).

In short, BeneRAG wins mainly by missing fewer questions that the model cannot answer on its own but whose retrieved context contains the answer.

W3. We are grateful for the suggestion to compare against strong existing routers; it made the evaluation considerably more convincing. Under our common protocol, both Adaptive-RAG and SKR retrieve more often than BeneRAG and still reach lower F1. Both differences are significant in paired tests (details in Q3, Part 3). We will add both methods to the main comparison figure and the appendix table, and cite SKR.

AuthorsOpenReview ↗

Response to Reviewer TesU

W4. Thank you for raising the cost question so directly. We agree with it. To answer it properly, we re-measured all methods under one common protocol: the same 500 questions, the same GPU, one query at a time, and end-to-end timing that includes routing, sampling, retrieval and generation (Table 16 in the revision).

MethodMean latency (s/query)F1Retrieval rate
Always RAG0.430.528100%
Q-B (question-only router)0.450.51671%
BeneRAG, parallel sampling0.960.51471%
BeneRAG, sequential sampling1.56——

On this benchmark BeneRAG is slower than both Always RAG and Q-B. Its F1 difference from either is not significant: −0.014 [−0.039, +0.011] vs Always RAG and −0.002 [−0.018, +0.012] vs Q-B.

We therefore do not claim compute efficiency, and the sentence you quote was meant to say exactly this. Our contribution is the supervision target: at the same retrieval budget, benefit supervision allocates retrieval better than correctness supervision. Efficiency follows only when retrieval calls themselves are costly or limited (break-even analysis in Q4).

Your question also made us check whether the observations are worth having at all. Over the question-only router they add a small but consistent gain at matched budgets:

Same budget30%50%70%
F-B − Q-B, Qwen2.5 (1 seed)+0.009+0.009+0.010
F-B − Q-B, Qwen3.5-9B (5 seeds, same sign in all)+0.011+0.010+0.005

We also checked what the observations contribute within a single question. We regenerated the sampled answers five times for each of 7,000 questions. Swapping BeneRAG's decisions between these repeated answers changes F1 by only 0.0004. On the questions whose label flips between repetitions, F-B is no better than F-C at telling which repetition will benefit. So the gain comes mainly from choosing which questions to retrieve for, not from reading one particular answer. Where compute matters, Q-B is the cheaper choice. We will state this in the Discussion.

Q1. Thank you again for this question; answering it taught us the most about our own method. We compared F-B with the correctness router F-C at the same retrieval rate (58.3%) and classified every error with fixed rules (no LLM judge):

ErrorCauseF-BF-CF1 contribution
Missed benefit (skipped, but retrieval would have helped)Knowledge gap, answer in context329690+0.0300
Knowledge gap, answer not in context187121−0.0017
Recall failure46115+0.0056
Answer-form mismatch81263+0.0067
Harmful call (retrieved, but retrieval hurt)Poor retrieval594520−0.0084
Reasoning error5632−0.0016
Conflicting evidence23+0.00001
Answer-form mismatch4417−0.0010
Totalincl. −0.00003 for skipped gains ≤ 0.05+0.0296
Overlap, not addedCorrect answer overwritten by retrieval378210−0.0113

Definitions: a knowledge gap means all 5 no-retrieval samples are wrong; a recall failure means the model knows the answer (at least 1 of 5 samples correct) but its final answer is wrong.

Almost all of the gain comes from the first row. F-B misses about half as many questions that the model cannot answer alone but whose context contains the answer (329 vs 690). It pays for this with somewhat more harmful calls on poor retrieval.

Mix shift: because the gain sits in that first row, it depends on how common such questions are. Reweighting that row from 0× to 2× moves the total from −0.0008 to +0.0472. Four further generators show the same pattern, with F-B − F-C between +0.013 and +0.035, all significant (GPT-4.1, Llama-3.1-8B, Phi-3.5-mini, EXAONE-3.5-2.4B).

We will summarize this in the Experiments section and place the full table in the appendix.

AuthorsOpenReview ↗

Response to Reviewer TesU

Q2. Thank you for asking about seed variance; it is a fair concern for gaps of this size. We retrained F-B and F-C with seeds 42–46 on the shared corpus. The table gives mean ± SD at exact budgets:

BudgetF-BF-CDifference
30%0.3926 ± 0.00210.3453 ± 0.0020+0.0473
50%0.4772 ± 0.00220.4332 ± 0.0009+0.0440
70%0.5388 ± 0.00180.5079 ± 0.0021+0.0310

F-B wins in all 5 seeds at every budget (paired t-test p ≤ 1.3×10⁻⁵).

At each router's own validation-selected threshold, the same five seeds give:

  • F-B over Always RAG: +0.010 ± 0.001 F1, significant in every seed.
  • F-B over F-C: +0.004 ± 0.001, positive in every seed but not significant within any single seed.

We will report the matched-budget gap as the robust result, and the operating-point gap to F-C as small.

Q3. We are grateful for the request to evaluate SKR and Adaptive-RAG under our protocol. Both methods use the same 10,000 test questions and the same answer branches as BeneRAG, and each method's threshold is tuned on validation UcallU_{call}.

Our pipeline has a single retrieval step, so we use Adaptive-RAG as a two-branch classifier (retrieve or not). It is a T5-large classifier trained with the official recipe on labels that ask whether the no-retrieval answer already matches the gold answer. Its multi-step branch has no counterpart in our setting.

MethodF1Retrieval rateBeneRAG − method, F1 [95% CI]
Adaptive-RAG (two-branch)0.40393%+0.012 [+0.007, +0.018]
SKR0.40075%+0.015 [+0.010, +0.021]
BeneRAG0.41558%—

Both differences remain significant after Holm correction. They come from a single training seed per method. BeneRAG retrieves less than both methods and still reaches higher F1.

Q4. Thank you for this question. It made us state the cost trade-off precisely rather than leaving it implicit. In the conditions we measured, no setting wins on both F1 and compute. Here generation cost is counted in output-token-equivalent units and retrieval calls are treated as free:

SettingF1: Always RAG → BeneRAGGeneration cost: Always RAG → BeneRAG
Canonical0.4000 → 0.414954 → 103
Shared corpus0.5532 → 0.5617140.3 → 183.9

BeneRAG skips retrieval for about 42% of questions, so the comparison reverses once each retrieval call costs more than about 120 units (canonical) or about 250–340 units (shared corpus). A paid search API could plausibly cost that much, but this is a calculation, not a measurement. We will state it in the Limitations.

Once again, we thank the reviewer for the time and care spent on our paper. These questions have made the paper more honest about where its contribution lies, and we would be glad to clarify anything further.

AuthorsOpenReview ↗

Update on Qwen3.5-9B: conclusions unchanged

Update: the answers above hold on Qwen3.5-9B. Reviewer yMwK's question about the generator backbone led us to move the whole paper to a newer generator, Qwen3.5-9B, and the paper is stronger for it: we regenerated all sampled answers, confidences and labels, retrained the routers with 5 seeds, and refined BeneRAG's loss to weight each question by max(|Δ|, 0.05), the F1 that a wrong decision costs (Δ = F1 with retrieval − F1 without). We therefore update the numbers from our answers to you below. The contribution is unchanged: benefit supervision allocates retrieval better than correctness supervision at the same budget. In the two main comparisons, fixed before a single test evaluation of BeneRAG, BeneRAG wins in all 5 seeds: +0.0152 [+0.0100, +0.0205] over Always RAG and +0.0406 [+0.0355, +0.0457] over the correctness router F-C at a 50% retrieval budget. Seed 42 unless noted; brackets are paired question-bootstrap 95% CIs.

W1. Same: the core claim holds in every environment we tested.

SettingBeneRAG − Always RAG (F1)BeneRAG − F-C, 30/50/70%
Canonical+0.015 (5 seeds)+0.054/+0.041/+0.018 (5 seeds)
No distractors−0.103 frozen, +0.002 retrained (n.s.)+0.035/+0.032/+0.012 retrained
Shared corpus−0.042 frozen, +0.004 retrained (n.s., 5 seeds)+0.047/+0.040/+0.024 (5 seeds)
BGE−0.088 frozen, −0.005 retrained+0.043/+0.036/+0.015 retrained

Unless marked n.s., all CIs exclude 0. Retrained, BeneRAG is above Always RAG on UcallU_{call} (F1 − 0.1 × retrieval rate) in all three shifted settings (+0.013, +0.020, +0.007).

W2, W3. Same; see Q1 and Q3 (SeaKR now included).

W4. Same position: we do not claim compute efficiency. Re-measured end to end (same 500 SQuAD questions, shared-corpus retrieval, one RTX 4090 on a different machine):

MethodLatency s/query, earlier → Qwen3.5F1Retrieval rate
Always RAG0.43 → 0.280.556100%
Q-B (question-only router)0.45 → 0.300.52367%
BeneRAG, parallel sampling0.96 → 0.690.51664%
BeneRAG, sequential sampling1.56 → 1.10——

BeneRAG's F1 is not significantly different from Q-B (−0.007 [−0.032, +0.017]). Unlike before, it is below Always RAG here (−0.040 [−0.069, −0.012]): this benchmark applies the canonical threshold to shared-corpus SQuAD, the frozen shared-corpus row of W1 (−0.042). The observations still add over the question at every budget:

Same budget, 30/50/70%Qwen3.5 (5 seeds)
BeneRAG − Q-B+0.009/+0.014/+0.015
BeneRAG − Q-B trained with the same weighted loss+0.012/+0.014/+0.012

Every per-seed CI is above 0. Swapping decisions across regenerated answers of the same question now lowers F1 by 0.0029 [0.0022, 0.0037] (3,500 questions); within a question BeneRAG is still not significantly better than F-C (+0.009 [−0.024, +0.042]). Where compute matters, Q-B is the cheaper choice.

Q1. Same pattern:

QuantityEarlier answerQwen3.5
Missed benefits, knowledge gap with answer in context (vs F-C)329 vs 690301 vs 534
Its F1 contribution / total+0.0300 / +0.0296+0.0213 / +0.0307
Mix shift: that row weighted 0× / 2×−0.0008 / +0.0472+0.0120 / +0.0422
Other generators, BeneRAG − F-C (all significant)+0.013 to +0.035 (4)+0.015 to +0.037 (5)

Even without that group, BeneRAG now leads F-C ([+0.0072, +0.0167]).

Q2. Shared corpus, BeneRAG and F-C retrained with seeds 42–46 (mean ± SD, exact budgets):

BudgetBeneRAGF-CDifference
30%0.4044 ± 0.00090.3575 ± 0.0015+0.0468 ± 0.0014
50%0.4951 ± 0.00200.4550 ± 0.0007+0.0401 ± 0.0014
70%0.5607 ± 0.00150.5363 ± 0.0007+0.0244 ± 0.0018

As before, BeneRAG wins in all 5 seeds at every budget, with every per-seed CI above 0 (p < 0.0001). At validation-selected τ, the gain over Always RAG is now smaller than before: +0.0040 ± 0.0018 (positive in all 5 seeds, p = 0.007; no per-seed CI excludes 0); over F-C, +0.0043 ± 0.0036 (positive in all 5; per-seed CI above 0 in 2 of 5).

Q3. Same:

MethodF1Retrieval rateBeneRAG − method
Adaptive-RAG (two-branch)0.42994%+0.012
SKR0.41977%+0.021
BeneRAG0.44059%—

All paired CIs exclude 0.

Q4. Same conclusion (generation cost in output-token-equivalent units, retrieval calls free):

SettingF1: Always RAG → BeneRAGCost: Always RAG → BeneRAGBreak-even per retrieval call
Canonical0.4283 → 0.440455 → 110≈135 (earlier ≈120)
Shared corpus0.5836 → 0.5878 (n.s.)141 → 192≈319 (earlier ≈250–340)

With one sampled answer instead of three, the canonical cost falls to 79 (break-even ≈73) at F1 0.4437.

Comments on the paper

4 comments

3. Revisions Addressing Reviewer Concerns (2/2)

Reviewer SWbv. The Qwen3.5-9B update restates every item below with the new numbers.

ConcernResponse (link)Revisions in the Manuscript
W1. What the estimator learnsW1Table 11; F.2 (question-conditioned controls, Tables 12 and F.1); Sec. 5
W2. Limited generalizationW2Cross-generator claim limited to ranking and matched budgets (Sec. 1, Sec. 4.4, Table 4, Table H.1); dataset transfer (Table 2, H.7)
W3. BaselinesW3Fig. 3 and Tables D.1–D.2 (SKR, Adaptive-RAG, SeaKR, Rowen)
W4. Narrow retrieval settingW4Sec. 4.4; H.3, H.5, H.8; Sec. 5 (scope of the retrieval settings)
Q1. Unseen datasetsQ1Table 2; H.7; weak ranking on NQ stated in Sec. 5
Q2. Published numbers of recent systemsQ2Sec. 2 (published results of SeaKR, Adaptive-RAG, SCAAR and LLM-independent adaptive RAG, each with its generator, subset and metric); BeneRAG's HotpotQA F1 in Table 2
Q3. Dense retrieval with BGEQ3Sec. 4.4; H.8, Tables H.4–H.5; query-specific pools, not full-corpus retrieval, stated in Sec. 5

Rowen, which was still running at the time of the update, is now in Fig. 3 and Table D.1 (F1 0.4199 at RR 0.784; BeneRAG 0.4404 at RR 0.594, seed 42).

Once again, we thank all three reviewers for these insightful questions. They have made our claims narrower but much better supported, and we would be glad to clarify anything further.

3. Revisions Addressing Reviewer Concerns (1/2)

During the discussion, several answers said that we would add, state or move a result. We applied these changes together in the revised manuscript. In Parts 3 and 4, the Response column links the reply with full detail, and the Revisions in the Manuscript column gives where each change appears.

Reviewer TesU. The Qwen3.5-9B update restates every item below with the new numbers.

ConcernResponse (link)Revisions in the Manuscript
W1. Coupling to the retrieval pipelineW1Abstract (gains depend on the protocol and can reverse under stronger retrieval); Sec. 4.4; Table 3; Tables 20–21; H.8; Sec. 5 (scope statement)
W2, Q1. Which error types produce the gainW2, Q1Sec. 4.3 (error paragraph); Tables I.1–I.4
W3, Q3. SKR and Adaptive-RAGW3, Q3Fig. 3; Tables D.1–D.2; SKR cited in Sec. 2
W4. Cost, and whether the observations matterW4Table 8 (BeneRAG vs Q-B at each budget); Table 11; F.2; G.5, Tables 15–16, Fig. 4; Sec. 5 (Q-B as the cheaper choice where compute matters)
Q2. Shared corpus with seed variationQ2G.4, Table G.1 (mean ± SD, seed test, per-seed intervals)
Q4. Joint gain in F1 and computeQ4Sec. 5 (stated as a limitation); G.5, Eq. G.1 (break-even calculation)

Reviewer yMwK

ConcernResponse (link)Revisions in the Manuscript
W1. Offline labeling costW1Sec. 5; G.5 (offline supervision cost)
W2. Transfer of the estimator to new datasetsW2Table 2; H.7, Table H.3
W3. Reliability of confidence and labelsW3F.1 (confidence calibration, confidence ablation, label stability); E.1
W4. Older backbone, small gainsW4Qwen3.5-9B throughout; weighted loss and its motivation (Sec. 1, Sec. 3); development subset excluded from evaluation (H.7); five-seed tests (G.4); new baselines (Fig. 3, Table D.1, D.4)
W5. Short-form QA onlyW5Sec. 5 (scope and future work)
Q1. Newer generatorQ1Qwen3.5-9B throughout; submitted Qwen2.5-7B source results kept for comparison (H.9, Table H.6); Qwen2.5-7B also evaluated as an unseen target (Tables 4, 17–18)
Q2. One threshold or one per datasetQ2Sec. 4.1 (one pooled threshold, validation UcallU_{call}); C.2 (per-dataset diagnostic)

In W4 we said we would post SeaKR and Rowen once they finished. Both are now in Fig. 3 and Table D.1: SeaKR reaches F1 0.4235 at RR 0.947, and Rowen reaches 0.4199 at RR 0.784, against 0.4404 at RR 0.594 for BeneRAG (seed 42).

2. Updated Experimental Results and Additional Evaluations

Main Contributions: Submitted vs. Revised

ContributionSubmittedRevised
C1. Benefit supervision improves retrieval allocationHigher F1 than correctness supervision at every tested budget; 3.6 F1 points at 50% (seed 42)Magnitude-weighted benefit supervision beats the correctness router at every tested budget; 4.06 F1 points at 50% (five seeds). The unweighted control still isolates the target (+3.7 points at 50%, five seeds)
C2. Semantic evaluation and conditions for deploymentBeats Always RAG with fewer retrieval calls; ranking transfers; F1 gains depend on the protocolGain over Always RAG holds in all five seeds; frozen transfer beats both fixed policies on three new datasets; the revised manuscript states that gains can reverse under stronger retrieval and that the observations add cost

Submitted numbers use Qwen2.5-7B and the unweighted F-B router (seed 42). Revised numbers use Qwen3.5-9B and BeneRAG. Five-seed values are means over training seeds 42–46. Brackets are 95% paired question-bootstrap intervals.

Results that were in the submission

ResultSubmittedRevisedWhere
Benefit minus correctness router, F1 at 30/50/70% budgets (seed 42)+0.049 / +0.036 / +0.014+0.052 / +0.041 / +0.020Fig. 2, Tables 8–9
Same, five seedsn/aBeneRAG +0.041 at 50%; unweighted F-B +0.056 / +0.037 / +0.007Abstract, G.4
Question-only benefit minus full-observation correctnessPositive at all budgetsSignificant at 30% and 50%; at 70% all five seeds are positive but the pooled interval includes zeroTable 8, D.1
F1 vs Always RAG at the validation threshold (seed 42)0.4149 vs 0.4000, RR 0.5830.4404 vs 0.4283, RR 0.594Fig. 3, Table D.1
Same, five seedsn/a0.4435 vs 0.4283, +0.0152 [+0.0100, +0.0205], RR 0.631Abstract, G.4
UcallU_{call} (seed 42)0.3566 vs 0.30000.3810 vs 0.3283Table D.1
Benefit AUROC0.7500.719 (unweighted F-B 0.757)Sec. 4.2, Tables 7 and 11
Semantic accuracy, GPT-4.1 judge, BeneRAG vs Always RAG0.509 vs 0.4680.525 vs 0.496 (mean of five seeds)Table 1
Agreement of F1 benefit labels with semantic labels89.1%91.4%E.1
Adapted SeaKR / Rowen, F10.392 / 0.4040.423 / 0.420Fig. 3, Table D.1
Shared corpus, minus Always RAG: frozen / retrained−0.017 / +0.0085; recalibrated +0.003−0.042 / +0.004 (interval includes zero); recalibrated +0.009Table 3, H.3
Distractor-free pools, frozen minus Always RAG−0.092−0.103; retrained +0.002 (interval includes zero)Tables 20–21
Generator transfer, frozen ΔF1 vs Always RAG: GPT-4.1 / Claude / Llama / Phi / EXAONE+0.010 / +0.011 / −0.022 / −0.007 / −0.019+0.004 / +0.010 / −0.020 / −0.011 / −0.021Table 4, Table 17
Generation cost per query (output-token units), BeneRAG K=3 / Always RAG103 / 54110 / 55Table 15

On GPT-4.1, the reference seed is inconclusive (+0.0041 [−0.0028, +0.0109]). Across five seeds the gain is +0.0085 ± 0.0037, positive in every seed (Table 17, H.1).

New in the revision

ResultValueWhere
SKR / Adaptive-RAG / information-need prompting, F1 (BeneRAG 0.4404)0.4193 / 0.4285 / 0.2849Fig. 3, Table D.1, D.4
Frozen dataset transfer, BeneRAG minus Always RAGTriviaQA +0.074, NQ +0.021, MuSiQue +0.030 (all intervals above zero); against F-C the result is mixedTable 2, H.7, Table H.3
BGE dense retrieval, minus Always RAG: frozen / retrained−0.088 / −0.005; the exact-budget gain over F-C remains (+0.043 / +0.036 / +0.015)Sec. 4.4, H.8
Execution benchmark, 500 questions: mean latency0.692 s (parallel) vs 0.284 s for Always RAG; F1 −0.0405 (interval excludes zero)Table 16, Fig. 4
Within-question decision exchange, 3,500 questionsOriginal decisions beat exchanged ones by +0.0029 F1 [+0.0022, +0.0037]F.2

Summary of Revisions and Updated Results

We sincerely thank the reviewers for their careful and constructive reviews. Their feedback led to substantial revisions to our methodology, experimental evaluation, and discussion of the limitations. We have uploaded a revised manuscript incorporating the experiments, analyses, and clarifications discussed in our responses. Every change is highlighted in red throughout the revised manuscript.

We organize this update into four parts. Part 1 summarizes the current method, main conclusion, and revised abstract. Part 2 compares the submitted and revised contributions and results, and presents additional evaluations. Parts 3 and 4 summarize each reviewer's concerns, provide links to our detailed responses in the OpenReview discussion, and identify the corresponding revisions and their locations in the revised manuscript.

1. Current Version and Main Conclusion

The revised manuscript uses Qwen3.5-9B as the source generator. BeneRAG denotes the final magnitude-weighted benefit-supervised method, while F-B and F-C denote the unweighted benefit- and correctness-supervised controls, respectively. Reference-seed results are distinguished from five-training-seed summaries.

Our central finding is that, under a limited retrieval invocation budget, learning the benefit of the deployed RAG branch leads to better retrieval allocation than learning whether the current no-retrieval answer is correct. Controlled comparisons isolate the supervision target by matching observations, model architecture, training loss, branch outputs, and retrieval budgets. Additional experiments across datasets, generators, and retrieval settings further examine the scope and limitations of this finding.

Importantly, better retrieval allocation at a fixed budget is distinct from achieving higher absolute F1 than Always RAG or reducing end-to-end inference cost. The former is supported by our controlled comparisons, whereas the latter outcomes depend on the retrieval protocol, operating point, and cost of obtaining pre-retrieval observations. The revised manuscript explicitly distinguishes these claims.

Key Revisions

  • Generator. Following yMwK (W4, Q1), the revised manuscript moves from Qwen2.5-7B to Qwen3.5-9B (thinking disabled), and all generator-dependent experiments were rerun.
  • Final router. BeneRAG now trains with magnitude-weighted benefit BCE. Each training question is weighted by max(|Δ|, 0.05), where Δ is the F1 change that retrieval causes, normalized by the training mean (Sec. 3). The unweighted benefit router (F-B) remains as a controlled ablation, so the supervision target and the weighting are tested separately (Fig. 2, Table 8).
  • Baselines on one protocol. SKR, Adaptive-RAG, SeaKR, Rowen and an information-need prompting router are evaluated on the same 10,000 test questions under the canonical retrieval protocol (Fig. 3, Tables D.1–D.2, Appendix D.4).
  • Generalization and cost. Frozen transfer to TriviaQA, Natural Questions and MuSiQue-Ans (Table 2, Appendix H.7); BGE dense retrieval (H.8); five training seeds (G.4); end-to-end execution cost (G.5).

Revised abstract

Answer correctness alone does not determine whether retrieval will improve a language model's response. Retrieval may leave an incorrect answer unimproved when it provides insufficient evidence. We study black-box selective retrieval through protocol-conditional context benefit, the improvement obtained by executing the deployed retrieval-augmented generation (RAG) branch. BeneRAG derives benefit labels and magnitude weights from paired branch outcomes and learns a routing score with magnitude-weighted binary cross-entropy. It predicts whether retrieval will improve the current answer using only the question, sampled no-retrieval answers, and verbalized confidence. At a 50% exact retrieval budget, BeneRAG exceeds the correctness router by 4.06 F1 points averaged over five training seeds. At validation-selected thresholds, its canonical five-seed mean F1 is 0.4435 versus 0.4283 for Always RAG, with a mean retrieval rate of 63.1%. The source-trained router also improves F1 over both fixed policies on TriviaQA, Natural Questions, and MuSiQue-Ans without target-side router fitting or threshold tuning. Automatic semantic evaluation with GPT-4.1 also favors BeneRAG over Always RAG. Gains over Always RAG remain protocol-dependent and can reverse under stronger retrieval, while acquiring the observations adds inference cost. These results support magnitude-weighted benefit supervision for allocating retrieval under a specified RAG protocol and budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.