- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether the sufficiency of retrieved evidence can be assessed from the question and evidence alone, before generating an answer. The authors assemble paired evaluation settings based on passage substitution, deletion, and question swapping, and audit construction shortcuts in these settings using 6 previous datasets. Then they propose RINSE, a lightweight scorer that combines content coverage, answer-span evidence, and a latent signal from a small language model, with evidence-size correction and equal-weight fusion. The method achieves an average paired concordance of 0.837, with a median latency of 36.5 ms, and is additionally evaluated on real retrieval outputs and in a pre-generation gating experiment.
Critical strengths
- The paper examines whether passage count, lexical overlap, and question-only/evidence-only signals predict the labels, highlighting the gap between distinguishing constructed positive/negative pairs and genuinely assessing evidence sufficiency.
- The lightweight scorer, RINSE, does not require answer decoding, and the combined scorer performs better on their modified datasets. Its performance good at accuracy and latency.
Critical weaknesses
- The methodological novelty is insufficiently established. Pre-generation sufficiency assessment and lightweight latent-sufficiency probes already exist. Table 4 largely compares adapted relevance or answer-confidence signals, leaving unclear whether RINSE’s fusion improves over a directly trained lightweight sufficiency classifier.
- The newly constructed negatives lack reported semantic validation. Replacing an annotated supporting passage does not guarantee that the remaining evidence is insufficient. The shortcut audits reveal substantial construction cues but do not establish label correctness, weakening the interpretation of the HotpotQA substitution results. RAG accuracy also affects the results, but it's unclear how your modified dataset affects the final answer. Due to there is no such analysis.
Refs:
[1] Xiaonan Li et al. LLatrieval: LLM-Verified Retrieval for Verifiable Generation. NAACL 2024. https://aclanthology.org/2024.naacl-long.305/
[2] Hailey Joren et al. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. ICLR 2025. https://openreview.net/forum?id=Jjr2Odj8DJ
[3] Roy Xie et al. Language Models (Mostly) Know When to Stop Reading. arXiv:2502.01025v2, 2025. The later work is titled Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signals. https://arxiv.org/abs/2502.01025
Questions for authors
- For the results showing in Table 1 what's the actual prompt and generating hyperparameters? These anlyisis must have a detailed disucussion and experiment to show this conclusion.
- The experimental setup lacks necessary declarations for environment, parameters, seeds, rounds, etc., and the baseline hyperparameter settings are also lacking, resulting in a significant lack of reproducibility.
- Appendix D uses the NeoQA validation set for calibration and top-5 validation. Are target validation labels permitted here, and which datasets determine layer 17, top-5, and calibration? While independent validation isn't test leakage, we must distinguish "excluding during training" from "ignoring for model selection."
- What is the added value of the three-reader fusion over a direct sufficiency classifier? Table 6 shows that the readers are complementary, but how would a simple sufficiency probe perform when using the same backbone, training splits, and top-five evidence input as RINSE's latent reader? If such a comparison is available, it would substantially clarify the methodological contribution beyond outperforming general relevance or answer-confidence signals.
- Were the new HotpotQA negatives independently checked for answerability? If so, what fraction remained answerable under each substitution scheme?
We thank Reviewer N19b for the helpful comments.
W1 & Q4. Prior work and sufficiency baselines. Section 2 and Appendix I now discuss LLatrieval, Sufficient Context, and Xie et al.’s hidden-state stopping probes. RINSE combines three lightweight signals to detect different problems with the evidence, rather than relying on an LLM verifier or a single model stream. To test whether combining these signals helps, we trained logistic and MLP probes to predict sufficiency using the same frozen Qwen2.5-1.5B layer-17 states, five-passage limit, and held-out splits as our latent reader. The probes score 0.759 and 0.753 on average. Adding the content and span readers raises these scores to 0.831 and 0.833, close to RINSE’s 0.837 (Section 6, Table 5, and Appendix E.1).
W2 & Q5. HotpotQA negative audit and impact on the real QA process. Appendix B.1 independently checks the substituted HotpotQA negatives using two answerability prompts with Qwen2.5-7B-Instruct. Both prompts flag 5.08% of random substitutions (209 of 4,117 pairs) and 6.05% of title-overlap substitutions (249 of 4,117 pairs). At least one prompt flags 11.78% and 13.55%, respectively. Removing pairs flagged by both prompts raises RINSE’s HotpotQA score from 0.853 to 0.858; removing those flagged by either prompt raises it to 0.862. Excluding HotpotQA entirely gives an average score of 0.834 across the remaining five datasets, compared with 0.756 for the strongest baseline (Section 3.3 and Table 6).
We also test how insufficient evidence affects QA while keeping the question and gold answer unchanged. When the model must answer, exact match drops from 31.0% to 6.0% on MuSiQue and from 50.0% to 35.0% on HotpotQA. F1 also drops from 44.1% to 13.2% and from 65.0% to 48.0%, respectively. On the swap pairs, exact match drops from 58.0% to 7.0% on SQuAD 2.0, from 69.0% to 18.0% on IIRC, and from 74.0% to 3.0% on fresh Wikipedia. These tests use Qwen2.5-7B-Instruct, with 200 pairs each for MuSiQue and HotpotQA and 100 swap pairs per dataset (Section 3.3 and Appendix B).
Q1, Q2 & Q3. Reproducibility. For Q3, Section 4 and Appendix D now explain how we used NeoQA validation to choose the main settings. Appendix E.2 adds a control that uses only the six training families. With the same five-passage limit, it scores 0.836 with layer 17 fixed and 0.834 when selecting a layer separately in each fold, compared with 0.837 for the main setup.
For Q1 and Q2, Appendix D and Table 8 now give the exact abstention prompt, chat-template settings, context limits, hardware, seeds, software versions, and reader revisions. Appendix D now specifies the chat-template and decoding settings, including disabling Qwen3 thinking and stopping at EOS or 24 new tokens. Appendices A, E.1, and F describe the baseline and training settings.