- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the distinction between evidence coverage and retrieval utility in retrieval-augmented generation (RAG). The authors construct controlled experiments on HotpotQA and MuSiQue by toggling the availability of gold evidence and comparing grounded and parametric generation. They derive an AUC upper bound for utility prediction using signals that are conditionally independent of utility given coverage, and evaluate reranker confidence and LLM-based context sufficiency as gating signals.
The main findings are that coverage alone is insufficient to predict whether retrieval improves generation, reranker confidence provides limited utility-prediction signal, and LLM-based sufficiency judgments sometimes perform better. However, the empirical improvements are small and inconsistent.
Critical strengths
- The authors report uncertainty estimates, multi-sample robustness checks, and several reproducibility issues encountered during experimentation.
Critical weaknesses
-
The paper has little novelty. The central observation that retrieving relevant evidence does not necessarily improve generation, particularly when the model can answer from its parametric knowledge, is well established. The main theoretical result is also rather straightforward.
-
A central motivation of the paper is that evidence coverage does not guarantee retrieval utility, partly because the model may already know the answer without retrieval. While this is a reasonable observation, the experiments do not sufficiently investigate the underlying causes of low retrieval utility.
Questions for authors
None