- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies adaptive RAG through the lens of evidence sufficiency. The authors identify a systematic mismatch between a language model's explicit sufficiency judgments and the sufficiency information encoded in its hidden representations: on controlled paired data, the native output is heavily biased toward predicting insufficiency, whereas a lightweight linear probe trained on the same final-layer representation achieves substantially better discrimination. Building on this observation, the authors propose SuREST, which recovers latent evidence sufficiency via a linear probe and uses it as a unified control signal for retrieval decisions. Evaluated on four multi-hop QA benchmarks, SuREST achieves the best macro F1 among all baselines while reducing retained passages.
Critical strengths
-
The identification of a representation–output mismatch for evidence sufficiency is conceptually compelling. The controlled paired-data construction and the careful comparison between the native LM head and a linear probe on the same hidden state (Table 1) convincingly demonstrate that this is a readout mismatch rather than an absence of information. The analysis of bias persistence across answer-correctness conditions further strengthens the claim.
-
Using a single latent sufficiency signal to jointly control retrieval triggering, evidence selection, and termination is a clean architectural choice. The SMG scoring function, which measures the reduction in latent insufficiency conditioned on the current evidence set, naturally discourages redundancy and favors complementary passages. This stands in contrast to approaches that combine multiple heterogeneous signals.
-
SuREST achieves the highest EM and F1 on all four datasets while substantially reducing retrieval cost (Table 2). The extensive ablations isolate the contribution of each component, providing clear evidence that the gains are not attributable to any single factor.
-
The probe transfers to Mistral and Qwen with competitive improvements and the linear-probe AUROC scales with model size (Figure 4).
Critical weaknesses
-
All experiments are conducted on Wikipedia-based multi-hop QA datasets. While these are standard benchmarks, the paper frames its contribution as broadly applicable to adaptive RAG. It remains unclear whether the latent sufficiency signal and the SMG selection strategy would transfer to other retrieval scenarios, such as single-hop fact verification, long-document QA, where evidence sufficiency may manifest differently in hidden representations.
-
Building the sufficiency probe requires substantial supervised effort: paired data construction with filtering, domain balancing, etc. This raises the question of whether the practical advantage over a simple fine-tuned classifier on sufficiency judgment is as large as the paper implies, especially given that the native-readout ablation requires offline calibration using the probe data.
-
Several relevant baselines are discussed but not directly compared, such as SetR, which explicitly selects passages as a set to cover complementary information. A direct comparison against SetR would help clarify whether the gains come from the sufficiency signal itself or from the state-dependent, set-aware selection that SMG implicitly provides.
Questions for authors
see weakness
3 comments
Response to Reviewer Wost (Part 2/2)
We compare the judge with SuREST without uncertainty-aware lookahead, since the judge lacks the five-seed ensemble required by this component.
Table c: QA performance and retrieval cost.
| Method | Macro F1 ↑ | Calls ↓ | Passages ↓ | Time (s) ↓ |
|---|---|---|---|---|
| SuREST (Full) | 33.25 | 3.53 | 6.44 | 7.99 |
| Probe w/o Lookahead | 32.35 | 3.58 | 6.02 | 7.16 |
| Explicit Judge (LoRA) | 31.28 | 2.47 | 4.64 | 5.28 |
Despite stronger offline discrimination, the judge achieves lower F1 than the probe without lookahead (31.28 vs. 32.35), with declines across all four benchmarks. Further analysis reveals more frequent sufficiency-based stopping by the judge: on MuSiQue, it stops retrieval in 397 cases, compared with 84 for the probe without lookahead.
The probe is also substantially cheaper to train. The LoRA judge has 40.37M trainable parameters and takes ~24 hours for training alone, exceeding the approximately 10 hours required for the probe's entire pipeline, including data preparation, feature extraction, and training. The five deployed L21 probe heads contain only 17,925 trainable parameters. Thus, while the judge achieves stronger offline discrimination and lower retrieval cost, the probe delivers higher end-to-end answer quality at substantially lower training cost. Detailed training configurations and implementation details are provided in Appendix C.3.2.
Finally, the calibration in Appendix C.3.1 applies only to the native-readout replacement ablation, not to standard SuREST. It uses offline probe scores to align native margins with the controller's thresholds and is distinct from the supervision-matched LoRA comparison above.
[W3] Comparison with SetR and Contribution Attribution.
We agree that SetR is a relevant baseline, as both methods aim to acquire complementary evidence, but their approaches differ. SetR selects a passage subset from a candidate pool based on explicit information requirements. In contrast, SuREST explicitly conditions SMG on the current evidence set and recomputes candidate gains after each addition, enabling incremental, state-dependent evidence acquisition. SuREST further uses the same latent sufficiency signal to control retrieval triggering and termination.
Our analyses provide evidence that both the latent sufficiency signal and state-dependent selection contribute to performance. The learned probe achieves stronger sufficiency discrimination than the native readout (AUROC 0.797 vs. 0.678; Table 1). Replacing SMG with BM25 rank-order selection reduces macro F1 from 33.25 to 31.52 (Table 3). Moreover, under identical evidence states and candidate pools, SMG selects missing supporting evidence more frequently than BM25 top-1 (10.91% vs. 8.37%; Table 5).
We agree that a direct comparison with SetR would further clarify the relative contributions of latent sufficiency and set-aware evidence selection, but time and resource constraints during the rebuttal phase prevent us from adding this baseline. We believe the current ablation studies and controlled evidence-selection results already provide evidence for the benefits of both components, although they do not fully isolate their individual effects.
Thank you very much for your reply. I keep my score of 3 unchanged.
Response to Reviewer Wost (Part 1/2)
We thank the reviewer for the insightful feedback and valuable comments!
[W1] Generalization Beyond Multi-Hop QA.
Thank you for highlighting this limitation. We additionally evaluated SuREST on 500 SQuAD v1.1 questions in the open-domain setting, without retraining the probe or changing the original backbone, thresholds, or retrieval budgets. Single-hop QA offers less room for adaptive evidence acquisition, as the initial BM25 results may already contain sufficient evidence. Nevertheless, SuREST improves F1 by 3.73 points over BM25 top-3 (paired-bootstrap 95% CI: [0.88, 6.60]). Compared with IRCoT, it achieves higher F1 while reducing retrieval calls, retained passages, and inference time by 35.9%, 61.2%, and 61.0%, respectively. These results support transfer beyond multi-hop QA, although SuREST incurs additional cost relative to fixed single-step retrieval.
Table a: Single-hop QA results on SQuAD v1.1.
We acknowledge that SQuAD remains Wikipedia-based and does not establish transfer to different retrieval domains. For long-document QA, a natural adaptation would be to split the target document into passages and restrict retrieval to that document. SuREST could then use the same sufficiency-guided selection and stopping mechanism. Extending this approach to different document structures and capturing evidence dependencies across distant sections are directions for our future work. We have added the single-hop evaluation and clarified these limitations in Appendix C.3.
[W2] Supervision Cost and Comparison with an Explicit Sufficiency Judge.
The probe supervision is derived from existing benchmark annotations rather than new manual sufficiency labels. HotPotQA pairs use annotated supporting facts, while MuSiQue provides paired answerable/unanswerable contexts (Appendix A.1). Filtering relies on automated answer checks with deterministic EM/F1 criteria, and domain balancing uses simple oversampling (Appendices A.1 and A.3). The backbone remains frozen during probe training. We acknowledge the one-time costs of data filtering and representation extraction.
To assess the practical trade-off, we additionally trained an explicit sufficiency judge by LoRA fine-tuning Qwen2.5-7B-Instruct using the same 6,200 unique training pairs, four prompt formulations, and two label mappings as the probe (99,200 training views; 156,800 examples per epoch after domain balancing). Its decision thresholds were independently calibrated using ground-truth labels from the frozen development set.
On the independent MuSiQue confirmatory set, the fine-tuned judge achieves stronger offline sufficiency discrimination.
Table b: Sufficiency discrimination on the MuSiQue confirmatory set.