acceptodds
Under review as a conference paper at ICLR 2027

Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG

Abstract

Retrieval-augmented generation (RAG) grounds large language models (LLMs) in external sources. However, retrieved passages often name the right entities while lacking the actual facts an answer requires. Generators rarely notice: even with an explicit instruction to abstain, 12 generators still answer 40.0–99.3% of insufficient-evidence questions. A common remedy trains abstention into the generator through fine-tuning or reinforcement learning. This ties the decision to one model's weights, potentially rewarding correct answers recalled from its parametric knowledge as if the evidence had supported them. The decision arrives only as the generator's own output, so every insufficient-evidence question still costs a full generator call. We ask instead whether sufficiency can be judged from the question and evidence alone, before any answer exists. We first demonstrate several pitfalls in constructing tests for insufficient evidence. Directly removing the relevant evidence or pairing evidence with unrelated questions can introduce cues that reveal the label without testing sufficiency, such as lexical overlap and evidence position. We therefore build a paired benchmark from substitution, deletion, and question-swap constructions that vary answer support while controlling selected surface features, such as word use. On this benchmark, sufficiency can be judged without generating an answer, but no single signal works on every dataset. We introduce RINSE (Relevance Is Not Sufficient Evidence), which combines three signals, one for each way evidence falls short: whether every part of the question is covered, whether any passage actually offers an answer, and whether a small language model, reading the passages together, treats them as sufficient. Across six datasets, RINSE ranks sufficient above insufficient evidence with a score of 0.837 (chance 0.5), ahead of the best of 10 prior methods (0.746) and of a frontier model asked the same question through an API (0.784). Its weakest dataset scores higher than any other method's weakest (0.684 vs. 0.676). It runs locally before generation, taking only 36.5 ms per question on a single GPU. Our code is available at https://anonymous.4open.science/r/RINSE-reproduction/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval-augmented generation, evidence sufficiency, abstention, hallucination, benchmark construction, shortcut cues

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_N19bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether the sufficiency of retrieved evidence can be assessed from the question and evidence alone, before generating an answer. The authors assemble paired evaluation settings based on passage substitution, deletion, and question swapping, and audit construction shortcuts in these settings using 6 previous datasets. Then they propose RINSE, a lightweight scorer that combines content coverage, answer-span evidence, and a latent signal from a small language model, with evidence-size correction and equal-weight fusion. The method achieves an average paired concordance of 0.837, with a median latency of 36.5 ms, and is additionally evaluated on real retrieval outputs and in a pre-generation gating experiment.

Critical strengths
  1. The paper examines whether passage count, lexical overlap, and question-only/evidence-only signals predict the labels, highlighting the gap between distinguishing constructed positive/negative pairs and genuinely assessing evidence sufficiency.
  2. The lightweight scorer, RINSE, does not require answer decoding, and the combined scorer performs better on their modified datasets. Its performance good at accuracy and latency.
Critical weaknesses
  1. The methodological novelty is insufficiently established. Pre-generation sufficiency assessment and lightweight latent-sufficiency probes already exist.[1,2,3]^{[1,2,3]} Table 4 largely compares adapted relevance or answer-confidence signals, leaving unclear whether RINSE’s fusion improves over a directly trained lightweight sufficiency classifier.[4]^{[4]}
  2. The newly constructed negatives lack reported semantic validation. Replacing an annotated supporting passage does not guarantee that the remaining evidence is insufficient. The shortcut audits reveal substantial construction cues but do not establish label correctness, weakening the interpretation of the HotpotQA substitution results. RAG accuracy also affects the results, but it's unclear how your modified dataset affects the final answer. Due to there is no such analysis.

Refs:

[1] Xiaonan Li et al. LLatrieval: LLM-Verified Retrieval for Verifiable Generation. NAACL 2024. https://aclanthology.org/2024.naacl-long.305/

[2] Hailey Joren et al. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. ICLR 2025. https://openreview.net/forum?id=Jjr2Odj8DJ

[3] Roy Xie et al. Language Models (Mostly) Know When to Stop Reading. arXiv:2502.01025v2, 2025. The later work is titled Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signals. https://arxiv.org/abs/2502.01025

Questions for authors
  1. For the results showing in Table 1 what's the actual prompt and generating hyperparameters? These anlyisis must have a detailed disucussion and experiment to show this conclusion.
  2. The experimental setup lacks necessary declarations for environment, parameters, seeds, rounds, etc., and the baseline hyperparameter settings are also lacking, resulting in a significant lack of reproducibility.
  3. Appendix D uses the NeoQA validation set for calibration and top-5 validation. Are target validation labels permitted here, and which datasets determine layer 17, top-5, and calibration? While independent validation isn't test leakage, we must distinguish "excluding during training" from "ignoring for model selection."
  4. What is the added value of the three-reader fusion over a direct sufficiency classifier? Table 6 shows that the readers are complementary, but how would a simple sufficiency probe perform when using the same backbone, training splits, and top-five evidence input as RINSE's latent reader? If such a comparison is available, it would substantially clarify the methodological contribution beyond outperforming general relevance or answer-confidence signals.
  5. Were the new HotpotQA negatives independently checked for answerability? If so, what fraction remained answerable under each substitution scheme?
1 comment

We thank Reviewer N19b for the helpful comments.

W1 & Q4. Prior work and sufficiency baselines. Section 2 and Appendix I now discuss LLatrieval, Sufficient Context, and Xie et al.’s hidden-state stopping probes. RINSE combines three lightweight signals to detect different problems with the evidence, rather than relying on an LLM verifier or a single model stream. To test whether combining these signals helps, we trained logistic and MLP probes to predict sufficiency using the same frozen Qwen2.5-1.5B layer-17 states, five-passage limit, and held-out splits as our latent reader. The probes score 0.759 and 0.753 on average. Adding the content and span readers raises these scores to 0.831 and 0.833, close to RINSE’s 0.837 (Section 6, Table 5, and Appendix E.1).

W2 & Q5. HotpotQA negative audit and impact on the real QA process. Appendix B.1 independently checks the substituted HotpotQA negatives using two answerability prompts with Qwen2.5-7B-Instruct. Both prompts flag 5.08% of random substitutions (209 of 4,117 pairs) and 6.05% of title-overlap substitutions (249 of 4,117 pairs). At least one prompt flags 11.78% and 13.55%, respectively. Removing pairs flagged by both prompts raises RINSE’s HotpotQA score from 0.853 to 0.858; removing those flagged by either prompt raises it to 0.862. Excluding HotpotQA entirely gives an average score of 0.834 across the remaining five datasets, compared with 0.756 for the strongest baseline (Section 3.3 and Table 6).

We also test how insufficient evidence affects QA while keeping the question and gold answer unchanged. When the model must answer, exact match drops from 31.0% to 6.0% on MuSiQue and from 50.0% to 35.0% on HotpotQA. F1 also drops from 44.1% to 13.2% and from 65.0% to 48.0%, respectively. On the swap pairs, exact match drops from 58.0% to 7.0% on SQuAD 2.0, from 69.0% to 18.0% on IIRC, and from 74.0% to 3.0% on fresh Wikipedia. These tests use Qwen2.5-7B-Instruct, with 200 pairs each for MuSiQue and HotpotQA and 100 swap pairs per dataset (Section 3.3 and Appendix B).

Q1, Q2 & Q3. Reproducibility. For Q3, Section 4 and Appendix D now explain how we used NeoQA validation to choose the main settings. Appendix E.2 adds a control that uses only the six training families. With the same five-passage limit, it scores 0.836 with layer 17 fixed and 0.834 when selecting a layer separately in each fold, compared with 0.837 for the main setup.

For Q1 and Q2, Appendix D and Table 8 now give the exact abstention prompt, chat-template settings, context limits, hardware, seeds, software versions, and reader revisions. Appendix D now specifies the chat-template and decoding settings, including disabling Qwen3 thinking and stopping at EOS or 24 new tokens. Appendices A, E.1, and F describe the baseline and training settings.

Reviewer_xrZ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a diagnostic benchmark to test retrieval-augmented generation systems' capabilities of detecting whether evidence is sufficient or not before generation, and then introduces RINCE, which combines (1) whether every part of a question is covered, (2) whether any passage provides evidence, and (3) whether a small LM judges it sufficient. Experimental results show RINCE outperforms Self-RAG and some other baselines on this diagnostic benchmark.

Critical strengths
  • This paper studies an important and interesting problem.
  • This paper provides a diagnostic benchmark to conduct a controlled study on this problem.
  • The proposed method shows effectiveness on the proposed benchmark.
Critical weaknesses
  • The method details aren't clearly specified. Although the method is conceptually simple, the paper omits important details, making it difficult to reproduce (e.g., the training objective and the size-correction fitting procedure). Some notation isn't clearly defined (e.g., α\alpha and β\beta).
  • Writing needs substantial improvement: some core terminology is introduced without being clearly explained.
  • The comparison may not be fair or clearly isolate the effectiveness of RINSE, and the comparison with Self-RAG is misleading: RINSE trains dedicated models for this task, while several compared baselines aren't specifically trained for it. Also, the claim of "outperforming Self-RAG" is misleading because the evaluated baseline is not the original Self-RAG inference method - Self-RAG uses a critic model only to label and generate training data for the final Self-RAG model.
  • The proposed RINCE score is designed for extractive QA tasks (eg reliance on extractive reader components), while today's RAG systems are used for open-ended generation tasks where the final answers aren't necessarily just extractive short spans. It is unclear whether the method is practically useful, especially since the main evaluations are on their diagnostic benchmarks.
Questions for authors
  • How can the RINCE score be applied to open-ended generation tasks? For instance, the original Self-RAG paper conducts evaluations on bio-generation tasks from FActScore. Can this method be applied to non-extractive or short-form QA tasks?
1 comment

We thank Reviewer xrZ4 for the helpful comments.

W1 & W2. Method details and exposition. Sections 3.1–3.3 now introduce the paired examples and explain how we evaluate predictions. We also clarify which cases of insufficient evidence each reader addresses in Section 4. In Section 4’s opening equation, α and β are the intercept and log-size slope fitted by ordinary least squares on the training pool. For the span and latent readers, we extend the fit beyond 22 passages using padded training records and a piecewise-linear fit. Appendices D and F give the fitting and training details.

W3. Baseline fairness and Self-RAG. We added logistic and MLP probes trained to predict sufficiency. They use the same frozen Qwen2.5-1.5B layer-17 states, five-passage limit, and held-out splits as our latent reader. Table 5 and Appendix E.1 report their results alongside fine-tuned cross-encoders. The probes score 0.759 and 0.753 on average, increasing to 0.831 and 0.833 when combined with the content and span readers.

We apologize for the incorrect “7B critic” wording. All experiments use the released inference model, selfrag_llama2_7b, as now clarified in Section 5 and Appendix A. Table 3 uses its IsRel score because that comparison evaluates sufficiency before answer generation. We also tested short-answer greedy decoding with IsRel, IsSup, and IsUse scoring on 1,200 pairs, 200 per dataset. Self-RAG with answer generation scores 0.722, compared with 0.762 for IsRel alone and 0.857 for RINSE. Median latencies on the 342-record timing benchmark are 463.4, 39.7, and 36.5 ms per record, respectively.

W4 & Q1. Non-extractive QA and open-ended generation. RINSE also handles questions whose answers are not text spans. It scores 0.934 on 1,382 numerical and yes/no pairs from IIRC, and 0.773 on NeoQA, where 99.6% of pairs require combining information from two articles (Section 6 and Appendix E.1).

On real retrieval outputs, using RINSE to choose between BM25 and dense top-five lists raises exact match from 39.3% to 48.9% and F1 from 49.8% to 61.7% when only one list contains sufficient support. Calibrated fallback also improves QA across the full query set, raising exact match from 36.9% to 38.1%, while calling a second retriever for only 28.0% of queries (Section 5 and Appendix H, Table 12).

RINSE checks whether passages contain enough information to answer a given question. In the FActScore dataset, the prompt asks for an introduction to a person without specifying what the biography should cover. We leave the assessment of such broad questions to future work.

Comments on the paper

1 comment

We thank Reviewer yVWc for the helpful comments.

W1, Q1 & Q4. Model choice. We added checks for construction artifacts and tested different layers, backbones, and rerankers. RINSE still performs well when passage count and word overlap are matched (Section 3.3, Table 2). It also scores 0.813 on 3,149 pairs from real BM25 and dense retrieval outputs (Section 5, Table 4).

With Qwen2.5-1.5B layers 16–28, RINSE scores 0.820–0.839. Using Qwen2.5-0.5B or Llama-3.2-1B gives 0.798 and 0.789. Fitting only on the six training families gives 0.836, compared with 0.837 for the main setup (Section 6 and Appendix E.2, Table 9). Replacing the multilingual reranker with the 23M English MiniLM-L6 gives 0.837 across the six datasets and 0.816 on real retrieval outputs, while reducing reranker p95 latency by 33% (Section 5 and Appendix E.3, Table 10). Our benchmarks and NQ-trained span reader focus on English QA, so testing other languages remains future work.

W2 & W4. Non-extractive reasoning, low lexical overlap, and other domain. In evaluations on held-out dataset families, RINSE scores 0.934 on 1,382 numerical and yes/no pairs from IIRC, and 0.773 on NeoQA, where 99.6% of pairs require combining information from two articles (Sections 5–6 and Appendix E.1). For each of the six datasets, we also test the 25% of pairs with the lowest word overlap between the question and evidence. RINSE averages 0.778, compared with 0.646–0.700 for individual readers and 0.700–0.701 for supervised classifiers using the same inputs. We leave controlled paraphrase tests and specialized domains beyond Wikipedia and news QA to future work.

W3, Q3 & Q5. Unfiltered retrieval, partial support, and entity distractors. Section 5 and Appendix G evaluate 1,500 unfiltered queries across five benchmarks, retrieving 5, 10, or 20 passages. In 40.9% of retrieved lists, the passages support only part of the reasoning needed to answer. Exact match is 6.8% with partial support, compared with 42.6% with full support. RINSE distinguishes full from incomplete support with pooled AUROC scores of 0.788, 0.776, and 0.746 at the three retrieval depths. Calibrated RRF fallback runs on 28.0% of queries and raises exact match from 36.9% to 38.1% (Appendix H, Table 12).

When two lists both lack required reasoning steps, both need additional evidence. Ranking one above the other does not resolve that gap. Size correction is designed to support a threshold at a fixed retrieval depth; it does not directly address ranking unpadded lists across depths. When lists from depths 5–20 are padded to match in length, RINSE scores 0.926.

We also add 20 near-duplicate distractors with changed entities to each of 400 groups. RINSE scores 0.805 and 0.745 in the symmetric and asymmetric settings, compared with 0.654 and 0.400 for the fine-tuned mDeBERTa classifier using the top five passages.

W2 & Q2. Gating accuracy and cost. On the balanced 600-record set in Table 1, converting RINSE scores to probabilities using the training folds gives an ECE of 0.056 and a Brier score of 0.217. At the threshold targeting 90% recall, precision is 0.560 and recall is 0.897. Across all 12 generators, gating blocks 20.0% of calls and reduces generation time by 11.2%. Answering on insufficient evidence falls from 63.6% to 51.1%, while answering on sufficient evidence also falls from 86.1% to 78.3% (Section 5 and Appendix D). Generation time falls by 14.1% for models below 2B, 10.2% for 4B–8B models, and 8.9% for 14B–32B models. Testing different thresholds shows how much computation can be saved at the cost of blocking some answerable queries.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.