acceptodds
Under review as a conference paper at ICLR 2027

Found but Forgotten: The Evidence Assembly Gap in Retrieval-Augmented Generation

Abstract

Building a strong retrieval-augmented answer generation system requires more than finding relevant passages: the system must also deliver the right combination of evidence to the reader. We began with a frozen HippoRAG2 core because it provides a strong graph-based retrieval foundation and exposes frontier traces that indicate where its search may be incomplete. Our goal was to determine how far this foundation could be pushed by improving the search-and-selection pipeline around it—and, in doing so, identify which bottlenecks remain once retrieval itself is already strong. We find that a candidate pool can contain all the evidence needed to answer a question, within the reader’s passage budget, yet the final context can still omit a crucial piece. We call this the evidence assembly gap. To study it, we separate three stages: whether sufficient evidence is available among the candidates, whether the complete evidence set is preserved during selection, and whether a bounded reader successfully uses it. A compositional-witness framework explains why passage-level recall does not guarantee recovery of a complete justification, and why simply giving the system more evidence need not improve its answers. We introduce CAST, a graph-assisted method built on top of a HippoRAG2 core with joint evidence selection. On 1,000 MuSiQue questions, the method improves token F1 by 11.89 points over the frozen native baseline and by 6.10 points over same-model pointwise selection, while keeping the reader fixed at five passages. Joint selection also reduces scenarios in which a feasible support set is available but discarded by 55.1%. Across seven packaged cohorts, whole-pipeline mean F1 improves on every dataset, with six nominal paired intervals above zero. Further analysis shows that preserving evidence and successfully answering are distinct problems, and that the value of additional reader context depends on how that context is constructed. Finally, model-assisted audit reveals different remaining bottlenecks across benchmarks: some exhibit near-complete annotated retrieval but substantial reference or scoring disagreement, whereas long-context QA continues to suffer from evidence-selection failures. Together, these results provide a concrete method for turning broad retrieval into compact, usable evidence and a stage-specific account of where further algorithmic and benchmark progress is needed due to dataset saturation and other issues, along with disaggregating different viable strategies to approach different types of retrieval problems in the future.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, retrieval, graphs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TUanOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes CAST to reduce evidence loss in RAG by expanding retrieval and selecting passages together.

Critical strengths

The main experiments show useful gains in answer quality and evidence preservation, and the controlled comparison is a clear strength.

Critical weaknesses
  1. The writing seriously limits understanding of the method. Dense explanations make the main contribution difficult to identify and assess.
  2. Without comparisons against closely related selection methods, I cannot tell how much CAST adds beyond existing work.
  3. The answer gains need further validation because the current output limit and answer extraction may affect the reported improvement.
Questions for authors
  1. How does CAST compare with SetR on identical candidate pools, and with OptiSet under matched retrieval and reading budgets?
  2. Do the gains remain with answers generated first and longer output limits, while keeping the selected passages unchanged?
  3. Does expansion using graph traces outperform expansion using only the question when retrieval calls, candidate counts, and reading budgets match?
Reviewer_pvuGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that RAG pipelines can lose answer quality after retrieval has worked: the candidate pool may contain every annotated support passage, but the bounded reader context drops one of them. The authors split evidence into three stages (found in the pool, preserved in the selected context, used by the reader) and propose CAST. CAST takes frontier traces from a HippoRAG2-derived graph walk, generates short corpus-wide dense+BM25 queries from them, fuses the hits with native candidates using duplicate-aware RRF, and then has an LLM jointly pick exactly k=5 passages for a fixed Qwen3.6-27B reader. On 1,000 MuSiQue questions, CAST improves F1 by 11.89 over the native final five and by 6.10 over a same-pool pointwise selector, and feasible-but-discarded support sets drop from 332 to 149. Whole-pipeline runs on six more cohorts show smaller gains, and the appendix adds an LLM-labeled audit of residual errors.

Critical strengths
  1. The controlled comparison in Table 1 / Table 2 is well designed for the specific question asked. Pointwise and joint selection see the same mixed cap-35 pool, use the same checkpoint, reader, and k=5, so the +6.10 F1 [3.73, 8.47] and +18.30 full-support gap really can be attributed to the selection protocol. I checked the partition counts in Table 2 and Table C1, the 61.12% → 82.55% conditional preservation, the 55.1% reduction, the per-stratum numbers in Table C4 (they reproduce 522/705 and the 9.29-point macro recall gap), and the k-endpoints in Table C6 against Table 3. They're all internally consistent.

  2. The found/over-budget/discarded/preserved partition (Section 2.2) is simple and useful, and the authors use it well. The point in Section 5.2 that native cap 10 looks better conditionally (575/613) only because it never exposes the hard cases is a good example of why opportunity and preservation have to be reported together, and the recall–completeness bound (Eq. 2) is elementary but makes the case for reporting full-set completeness instead of average recall.

  3. The paper reports its own negative results instead of hiding them. Native protection and the cap-35→50 change show no F1 benefit (Table 3), CAST's k=5→8 gain is null while native ranking gains +3.04, 75% of the joint-vs-pointwise gain comes from answer-marker transitions (Table C3), and the 507 both-complete questions show −1.29 F1 for joint selection. Limitations (Section 7) is candid about what is bundled and what isn't controlled.

Critical weaknesses
  1. Much of the headline gain looks like a reader-completion effect. Per Table C3, the Answer: marker is missing on 246/1,000 pointwise and 192/1,000 joint outputs, and marker-discordant cases account for 4.57 of the 6.10 points. On the 507 both-complete questions, joint selection is actually worse (−1.29), so I'd want a rerun under an answer-first or larger-budget contract.

  2. The frontier-search component, CAST's named contribution, isn't shown to matter. The generated-only arm reaches 60.00 vs. 60.48 for the mixed pool (Table C2), and joint selection over native top-20 alone recovers +8.72 of the +11.89 (Table 1). Without a question-only multi-query expansion control at matched cost, trace conditioning stays unvalidated.

  3. The baselines and scope are narrow. Pointwise 0–9 scoring with position tie-breaking is a weak reranker, and no listwise reranker or cited set selector (SetR, DPS, OptiSet) is run. All controls use one MuSiQue cohort and one model. NQ and PopQA gains fail sign tests (p=0.880, 0.289; B.4).

Questions for authors
  1. What fraction of reader outputs hit the 1,024-token ceiling in each arm? If you rerun native-five, pointwise, and joint with answer-first prompting or a larger output budget, how much of the 6.10 and 11.89 gaps remains?
  2. Can you run a question-only multi-query expansion baseline (same number of queries, same dense+BM25 fusion, same caps, same joint selector)? That's the comparison that would justify the "search traces" part of CAST.
  3. How often do pointwise scores tie at the top-5 boundary, and how does pointwise do with a finer scale, or with logit-based scores instead of generated integers? How does a listwise reranker compare on the same pool?
Reviewer_xJxROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the evidence assembly gap in retrieval-augmented generation, where the candidate pool contains all annotated supporting passages, but context selection fails to preserve the complete evidence set despite a sufficient passage budget. The authors propose CAST, which generates additional search queries from HippoRAG2-derived graph retrieval traces, combines newly retrieved passages with the original candidates, and selects a fixed number of passages for the downstream reader. Based on seven QA benchmark's subsets, the authors show that CAST outperforms all compared baselines.

Critical strengths
  1. The paper provides a useful perspective on RAG failures by distinguishing evidence discovery, preservation, and use. In particular, it highlights cases where all supporting passages are available in the candidate pool but are not preserved during context selection despite a sufficient passage budget.

  2. The proposed method CAST demonstrates meaningful performance gains over the evaluated baselines in controlled experiments, particularly over pointwise selection with the same candidate pool and reader passage budget.

Critical weaknesses
  1. The proposed method primarily combines established components, including graph-based retrieval, query expansion, and joint passage selection. Its main methodological distinction appears to be conditioning query expansion on HippoRAG2-derived retrieval traces. However, the experiments do not isolate the benefit of this design over simpler query expansion or iterative retrieval approaches. It is unclear what is the technical contributions.

  2. The proposed method CAST uses additional query expansion and retrieval, but the evaluation lacks budget-matched iterative retrieval baselines, such as IRCoT. However, it remains unclear whether the gains from search expansion arise from the proposed trace-conditioned queries or simply from additional retrieval effort.

Questions for authors
  1. The presentation would benefit from a clearer separation between the problem formulation, the methodological contribution, and the empirical findings. The current narrative is heavily centered on experimental comparisons, making it difficult to identify the core technical contribution and understand why each component is needed.

  2. Could the authors provide more targeted analyses to clarify the contribution of individual components, including budget-matched ablations of trace-conditioned versus question-only query expansion?

  3. The experimental results are all presented in tables. Visualizations, such as plots, are very important for readers to quickly understand the contents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.