acceptodds
Withdrawn from ICLR 2027

Retrieval Gains That Do Not Reach the Reader: Why Recall Misleads in Both Directions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Retrieval-augmented generation is often evaluated by retrieval quality (e.g., Recall@), implicitly assuming that better retrieval yields better answers. We test this assumption directly. We introduce SCOPE (Schema-Checked One-Pass Expansion), which extends SIRA's validated query expansion to long structured documents (financial filings and papers). In a single call, SCOPE proposes additional search terms   and  likely table locations for the answer, each proposal is validated against the document before one search is issued. On questions from UDA, SCOPE improves Recall@10 over SIRA by points, and by points when the answer is in a table. Its retrieval is statistically indistinguishable from a five-turn ReAct agent (, with interval ) while using over fewer search tokens, and a one-turn agent matches it more cheaply still. In filings, much of the gain comes from a single instruction: preserve every year and fiscal period, since the year is often not the answer but the column it appears in. Despite delivering correct evidence more often, SCOPE does not measurably change answer accuracy. The net effect is churn, and evidence arrives for questions but disappears for that were previously answered well (), while newly served questions are answered worse (). Aggregate recall hides these transitions and can mislead in the other direction. At the same time, a re-ranker that reduces SCOPE by 5.4 Recall@10 points causes no measurable accuracy loss. A five-search agent improves accuracy with no measurable recall gain by collecting more varied context. We therefore recommend reporting evidence transitions (which questions gain vs. lose evidence) and accuracy conditional on evidence, alongside Recall@. We include document-level intervals throughout and report our preregistered targets and negative results in full.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval augmented generation, query expansion, evaluation, document question answering, retrieval metrics, structured documents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_m7gKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SCOPE, a new retrieval system that targets financial filings and papers by proposing a series of components based on properties of such long structured documents. Experiments report that the method has a moderate gain in retrieval performance but no improvement in answer quality. The paper is being honest in reporting a major source of the gain, which is not from the proposed components but from a prompt change discovered from the test set.

Critical strengths

None

Critical weaknesses
  1. Methodology-wise, the paper is trivial. The proposed components are not validated by experimental gains, and the significant gains come from "A one-line rule for query modification, to keep every year and fiscal period." which carries no scientific value at all.

  2. Experiment-wise, the core claim is weird. While the paper has a diagnostic paper-style title, it reads to me like a methodology paper at the beginning, and it indeed describes a new method. However, when we arrive at the experiments, the conclusion of the experiments was that the gains are from an instruction that was discovered "post hoc" in the test set, and that the gains do not improve the answers in RAG. While negative methods could be potentially helpful, I don't see any value of a paper to propose a random method and use experiments to prove that this method does not work at all. I thank the real author of this paper for being honest.

Questions for authors

What is the main claim of this paper?

Reviewer_jST3OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SCOPE, a single-pass retrieval method that extends SIRA with query rewriting and schema-validated structural cues for retrieving evidence from long, structured documents. The authors evaluate SCOPE on 539 questions from three UDA subsets and report improvements in Recall@10 over vanilla SIRA. However, these retrieval gains do not translate into measurable improvements in downstream QA performance. Through question-level evidence transition analysis, the authors demonstrate how aggregate retrieval metrics can obscure changes in evidence availability and answer quality.

Overall, the paper addresses an important problem in RAG evaluation and provides useful empirical insights into the relationship between retrieval quality and downstream performance.

Critical strengths
  1. The paper investigates an important limitation of retrieval metrics in RAG systems. Demonstrating that improvements in Recall@k do not necessarily translate into better downstream answers is valuable.

  2. The question-level evidence transition analysis provides useful insights beyond aggregate retrieval metrics. By examining questions that gain or lose evidence between retrieval methods, the paper offers a more detailed explanation of why retrieval improvements may not lead to corresponding QA gains.

  3. The authors transparently report negative results and unsuccessful preregistered targets, which strengthens the credibility of the empirical analysis.

Critical weaknesses
  1. The methodological novelty of SCOPE looks limited, since much of its improvement comes from query rewriting rather than the proposed structural validation mechanism. Appendix E shows that adding structural information to the rewrite-only configuration improves overall Recall@10 by only 0.016, with a confidence interval spanning zero. Although the improvement is more substantial on table-heavy questions, the contribution of structural validation to the full method remains unclear.

  2. Relevance labels are automatically derived from UDA's evidence spans rather than manually verified, and only 460 of the 539 test questions have locatable evidence. Since the central conclusions depend on identifying which questions gain or lose evidence, potential labeling errors could affect the analysis. Furthermore, the main downstream QA experiments use only one reader model, limiting the generality of the findings.

Questions for authors
  1. Given that structural validation provides only a small overall improvement beyond query rewriting, what is its main contribution to SCOPE? Can the authors further clarify the benefits of structural validation independently of query rewriting?

  2. How reliable are the automatically derived evidence labels, particularly for the 49 questions gaining evidence and 18 questions losing evidence?

Reviewer_XpshOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposed a new retrieval method that combines query expansion, BM25, and structural keys. It leads to improved recall on financial filings type of corpus compared to vanilla BM25, dense retrieval, and react style agentic retrieval. However, the gain in retrieval recall does not lead to answer accuracy improvement.

Critical strengths
  1. Design customized retrieval techniques are a specific domain is useful and important in practice.
Critical weaknesses
  1. The paper is hard to comprehend. A lot of expressions are vague and hard to understand.

line 64-75, this whole paragraph is unclear. especially: "Answer accuracy does not measurably change because, per question, the net delivery gain is evidence moving both into and out of the window, and the questions that lose it are the ones the reader was answering best, and what the newly reached questions add is drained away again."

line 50 “Agents attack this by searching” and line 96 "learned sparse representations (Formal et al., 2021) attack the mismatch from the index." I'm not sure "attack" is the right word to use here.

The abstract talked about SIRA and lots of detailed numbers without giving the reader enough context.

“a strong enough instrument for the measurement to mean something” vague and hand-wavy

  1. The paper starts by stating that "Retrieval-augmented generation is often evaluated by retrieval quality (e.g., Recall@k)". However, this is a wrong statement. RAG is usually evaluated by the end performance, e.g., QA performance, not by the retrieval performance.

  2. Some design choices are very specific to financial filings, e.g., "A one-line rule for query modification, to keep every year and fiscal period."

  3. In general, the retrieval method is incremental and the conclusion that better recall does not always lead to better answer is already known.

Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.