acceptodds
Under review as a conference paper at ICLR 2027

Coverage Is Not Utility: An Exact Ceiling on Selective Retrieval

Abstract

Retrieval-augmented generation degrades when the corpus does not cover the query, and the usual remedy is a gate that estimates retrieval confidence and falls back to parametric generation. Coverage is a useful intermediate variable, but it does not determine whether retrieval helps. Using HotpotQA-distractor and MuSiQue, which ship gold and distractor paragraphs for every question, we tog- gle evidence availability while holding question, distractors and pipeline fixed, obtaining ground truth for both coverage (is the evidence indexed?) and utility (does grounding beat answering from parameters?). The two diverge: grounding helps on only 29.4% of questions whose evidence is indexed. Because we con- trol coverage, we can evaluate a gate that knows it exactly, and it reaches only AUC 0.698 at predicting utility on HotpotQA and 0.757 on MuSiQue. This ceil- ing binds every score that carries no information about utility beyond coverage, cannot be raised by recalibration, and decomposes in closed form into two rates measurable without building any gate: how often grounding helps when evidence is and is not indexed. Reranker confidence, the standard signal, detects coverage (AUC 0.734) but predicts utility at only 0.602–0.675 and carries no detectable utility signal within the covered set, so the ceiling binds it. Asking the genera- tor whether the passages suffice does carry such signal on HotpotQA and reaches 0.639–0.724, though its advantage over the reranker is significant on one of three backbones and absent on MuSiQue; we offer it as an existence proof, not a better gate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, selective retrieval, gating, adaptive retrieval, evaluation, analysis, question answering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_yaEaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the distinction between evidence coverage and retrieval utility in retrieval-augmented generation (RAG). The authors construct controlled experiments on HotpotQA and MuSiQue by toggling the availability of gold evidence and comparing grounded and parametric generation. They derive an AUC upper bound for utility prediction using signals that are conditionally independent of utility given coverage, and evaluate reranker confidence and LLM-based context sufficiency as gating signals.

The main findings are that coverage alone is insufficient to predict whether retrieval improves generation, reranker confidence provides limited utility-prediction signal, and LLM-based sufficiency judgments sometimes perform better. However, the empirical improvements are small and inconsistent.

Critical strengths
  1. The authors report uncertainty estimates, multi-sample robustness checks, and several reproducibility issues encountered during experimentation.
Critical weaknesses
  1. The paper has little novelty. The central observation that retrieving relevant evidence does not necessarily improve generation, particularly when the model can answer from its parametric knowledge, is well established. The main theoretical result is also rather straightforward.

  2. A central motivation of the paper is that evidence coverage does not guarantee retrieval utility, partly because the model may already know the answer without retrieval. While this is a reasonable observation, the experiments do not sufficiently investigate the underlying causes of low retrieval utility.

Questions for authors

None

Reviewer_GoPNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes corpus coverage—whether supporting evidence is indexed—from retrieval utility—whether grounding improves the answer relative to parametric generation. Using paired GOLD/NOGOLD conditions on 500-question HotpotQA-distractor and MuSiQue sets, the authors measure both quantities while holding questions, distractors, and the answering pipeline fixed. They derive an AUC upper bound for any utility-prediction score that is conditionally independent of utility given coverage, expressed in terms of the utility rates in the covered and uncovered strata. The experiments show that reranker confidence is more predictive of coverage than of utility. A generator-based context-sufficiency score exhibits additional within-covered utility signal on HotpotQA, but its advantage over reranker confidence is statistically significant on only one of three backbones and does not extend to MuSiQue. The paper frames these results as a diagnosis of coverage-based gating, not as evidence for a consistently superior retrieval policy.

Critical strengths
  1. Meaningful practical empirical gains with compact inference interface. The proposed system yields large F1 improvements (+3.8 pooled points) versus uncompressed context on 2WikiMultiHopQA and MuSiQue, cutting reader‑side token consumption by ~90% under only one additional LLM call.

  2. Important real‑world multi‑hop evidence‑preservation problem is highlighted. Simply keeping answer sentences does not guarantee multi‑hop intermediate bridge evidence survives compression. The paper proposes chain‑structured output plus‑bridge‑aware training as a concrete solution. If the underlying mechanism receives more direct validation, this work can offer insights beyond plain aggregate answer‑accuracy numbers.

Critical weaknesses
  1. Over‑reaching theoretical claims plus algebraic bug in Theorem B.6. Theorem B.6 contains an algebraic error: factor rr cancels in expected benefit a⋅r⋅Ga\cdot r\cdot G and cost c⋅rc\cdot r, giving threshold a≥c/Ga\ge c/G instead of a≥c/(rG)a\ge c/(rG). Theorem B.5 is merely a cost‑accounting identity and cannot prove single‑call compression is necessary. Theorem B.2 only holds under full‑signal‑preservation assumption and is not a universal upper bound for lossy abstractive compressors. Yet the paper draws strong conclusions, rejecting alternative designs and claiming a saturated compression frontier based on these results.

  2. Missing causal ablations, baseline comparisons and inconsistent robustness results. The paper lacks matched‑compute ablations to disentangle contributions from chain‑format SFT, unfiltered self‑training, bridge‑only filtering and semantic‑only filtering. Head‑to‑head comparisons against related compressors (BRIEF, RECOMP, EXIT, CompAct) under identical reader/token budget are absent. Robustness sweeps in Figure 3 show multi‑point performance fluctuations, conflicting with text claims of changes “at most 1.1 points” and “no cliff”. Only the full end‑to‑end system is validated; we cannot confirm bridge‑survival filtering drives gains nor consistent cross‑setting robustness.

Questions for authors
  1. Your coverage‑based AUC ceiling rests on the conditional independence assumption s⊥y∣cs \perp y \mid c. Right now your supporting evidence is only that within‑coverage‑stratum AUC confidence intervals contain 0.5. Have you analysed utility‑signal behaviour inside the \textit{uncovered} question stratum, including possible non‑monotonic trends or predictable signal? If you have not done this analysis, would you adjust your manuscript wording to a more modest claim: “there is no statistically detectable ranking signal within covered examples”, rather than stating reranker scores satisfy this strict conditional‑independence theoretical assumption?

  2. Could you state the precise evaluation metric plus tie‑breaking / normalization rules you use to construct binary utility labels (y=1y=1 when retrieval grounding helps)? Using this exact same metric, please report the fraction of help‑tie‑harm outcomes. Also provide absolute performance or regret of the optimal coverage‑only routing policy compared to the utility oracle. These numbers help us judge whether the reported AUC ceiling creates meaningful practical limits for real‑world routing.

  3. Please report what fraction of GOLD‑setting questions have the full annotated gold supporting evidence contained within top‑3 retrieved passages, broken down per dataset and per hop count. If feasible, also check whether the core coverage‑utility gap still persists for the subset of questions where top‑3 passages already hold all gold supporting evidence. This helps us tell whether the observed gap comes from fundamental coverage‑utility tension or quirks of your particular retrieval setup.

Reviewer_ANc4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the difference between having answer evidence in the index and getting a better answer by using retrieval. The authors add or remove supporting passages on HotpotQA and MuSiQue. They give an AUC bound for scores that use coverage alone to predict retrieval benefits, then compare reranker confidence with a model-based sufficiency score. The sufficiency score helps in some settings, but its advantage is not consistent.

Critical strengths
  1. The paired GOLD/NOGOLD setup allows a direct comparison between evidence availability and answer improvement. Using the same candidate answers across policies also makes the comparison fairer.
  2. The bound is easy to interpret. The authors test several models, a second dataset, and multiple answer samples. They also clearly report results that are not statistically significant.
Critical weaknesses
  1. Prior work has already studied how model knowledge affects retrieval benefits and why enough context does not always lead to a correct answer. This paper adds controlled tests and a conditional AUC bound, but the new insight and practical value remain limited.
  2. A within-group AUC that is not significantly different from 0.5 does not prove the conditional independence required by the bound. The experiments also use older model families, lack tests on newer models of similar size, and lack comparisons with existing knowledge-aware retrieval methods. These gaps limit how broadly the findings apply.
Questions for authors
  1. What important new insight does the bound add to earlier work on model knowledge and context sufficiency? How can it guide the testing or design of retrieval gates?
  2. What evidence supports the conditional-independence assumption beyond an AUC that is not significantly different from 0.5? Should the claim about the reranker be narrowed?
  3. Can the main findings be tested with a newer model of similar size and an existing knowledge-aware retrieval method that fits the same setup?
Reviewer_WbaJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether having the right evidence available actually makes retrieval useful. It adds or removes gold evidence on HotpotQA and MuSiQue, then compares answering with retrieval against answering without it. It derives an AUC score bound for scores that only reveal evidence availability and compares reranker confidence with the model’s own assessment of the passages. According to the paper's results - neither factor seems to reliably predict when retrieval helps.

Critical strengths

The main strength I see, in principle, is that the paired design separates whether evidence is available from whether it actually helps. Reusing the same candidate answers also isolates the routing decision from generation variability. This could support a useful analysis, though the issues below limit what can be concluded from it.

Critical weaknesses
  • The paper has not shown that its main bound applies to the reranker. The bound requires the score to contain no additional information about retrieval utility once coverage is known. An AUC interval containing 0.5 does not establish this. Even an AUC of exactly 0.5 can hide predictive information. For example, when both high and low scores indicate that retrieval helps. Cases without gold evidence are not tested either. The proof is elementary, as the authors acknowledge, and the empirical argument needed to make it consequential is missing.
  • The analysis contains basic errors that undermine confidence in its conclusions. The introduction attributes the lack of retrieval benefit mainly to the model already knowing the answer. Yet accuracy without retrieval is only 17.8%, and Table 1 implies that both answers are wrong on 52.8% of covered questions. The more common outcome is failure either way. Section 5 also incorrectly states that probabilities below 0.1 become zero when estimated from ten samples. -The authors disclose using AI for drafting and analysis, but basic errors along with the use of other "AI slop terminology" and contradictory information throughout the paper should merit a rejection.
Questions for authors
  • Why does an AUC interval containing 0.5 justify assuming independence? What about cases without gold evidence, which were not tested?
  • Why does the paper claim that retrieval adds nothing because the model already knows the answer when both answers are wrong on 52.8% of covered questions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.