BeneRAG: Benefit Supervision for Black-Box Selective Retrieval

Answer correctness alone does not determine whether retrieval will improve a language model's response. Retrieval may leave an incorrect answer unimproved when it provides insufficient evidence. We study black-box selective retrieval through protocol-conditional context benefit, the improvement obtained by executing the deployed retrieval-augmented generation (RAG) branch. BeneRAG learns from paired branch outcomes offline and predicts whether retrieval will improve the current answer using only the question, sampled no-retrieval answers, and verbalized confidence. With observations, architecture, and training held fixed, benefit supervision yields higher F1 than correctness supervision at every tested budget in exact-budget ranking evaluations, gaining $3.6$ F1 points at a $50$% invocation rate. A question-only benefit estimator also outperforms the full-observation correctness estimator at these budgets. Across three QA datasets with controlled evidence availability, the policy improves answer quality over Always RAG while invoking retrieval less often. Automatic semantic evaluation with GPT-4.1 also favors BeneRAG over Always RAG. Benefit ranking remains useful under corpus and generator changes, but quality gains depend on the protocol and operating point. Acquiring the observations also incurs additional execution cost. These results support supervising the incremental benefit of the available RAG branch when allocating a limited retrieval budget.