acceptodds
Under review as a conference paper at ICLR 2027

What Retrieval Buys Depends on Who Reads: Splitting Multi-Hop RAG Accuracy into Assembly and Reading

Abstract

Multi-hop questions name their endpoints but not the passages that connect them. Retrieval-augmented systems reach those passages either by building an entity graph offline or by retrieving iteratively with a model in the loop, and the two are compared by end-to-end accuracy. Accuracy, however, cannot tell whether a system failed to assemble the evidence or its reader failed to use it. We separate the two steps with two controls: at a matched passage count only the retrieval varies, and with identical passages only the reader varies. Whether the evidence holds the whole chain is scored apart from the answer. Together the controls split each reader's accuracy gap to gold evidence into an assembly share and a reading share. Measuring the steps apart changes verdicts. On MuSiQue, end-to-end accuracy ranks a single read of a graph index at least as high as an iterative loop. Yet at the same passage count the loop completes far more chains, and its passages raise accuracy once read after the search. The split also shows that the bottleneck differs by reader. On MuSiQue's deep questions most of the 8B readers' gap is reading: the index's top-ranked wrong passages cost open readers up to 32B more than the frontier model, at least twice as much by point estimate. Training moves one share and not the other: a small model distilled call by call from the loop's search learns to search, with no detectable reduction in its reading share. Running alone, it outscores the published systems we compare on the same MuSiQue questions. Retrieval gains are thus relative to the reader, and we propose evaluating them at a matched passage count and per reader.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval-augmented generation, multi-hop question answering, graph-based RAG, agentic retrieval, evidence completeness, retrieval noise robustness, knowledge distillation, controlled evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kp9tOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce an analysis for multi-hop rag problem. They do a detailed analysis of this problem in two parts: Retrieval and Reasoning. Then they do a small SFT Experiment to improve the ability for solving multi-hop rag problem.

Critical strengths

They divided the multi-hop RAG problem into two tiers for analysis. The experiments in this part were very solid, comparing mainstream baselines.

Critical weaknesses

The first half of the paper points out one of the main bottlenecks for small models is reading distractor evidence. However, the second half mainly tests the difference between searching and reading using search trajectory distillation, without further showing how to use this diagnosis to fix the reading bottleneck. Therefore, I feel the SFT part is mostly just validating their analysis, and its contribution as a method for performance improvement is still limited.

Questions for authors

The distillation experiments only evaluated the Opus 4.6 to Qwen3-8B-Base teacher-student pair. I want to see a broader range of student models across different sizes and model families.

Reviewer_ZL83OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an evaluation framework that separates multi-hop RAG performance into evidence assembly and reading. The authors compare HippoRAG 2 with iterative retrieval and investigate how different language models utilize retrieved evidence. The paper also explores distilling iterative retrieval capabilities from a frontier model into a smaller model.

Critical strengths
  1. The paper addresses an important problem in understanding retrieval and reasoning failures in RAG systems.

  2. The experiments provide some interesting observations about how different language models are affected by distracting passages.

Critical weaknesses
  1. Limited novelty and insufficient discussion of related work. The proposed decomposition of RAG performance into assembly and reading is relatively straightforward, relying on comparisons between gold evidence, gold evidence with distractors, and retrieved passages. Previous studies have already investigated retrieval effectiveness, generation performance, and the interaction between retrieved information and language models [1, 2, 3]. The authors should clarify how their evaluation provides fundamentally new insights beyond these existing studies.

  2. Insufficient evaluation across retrieval methods. The paper only evaluates HippoRAG 2 as the graph-based retrieval method, which is insufficient to support general conclusions about graph-based versus iterative retrieval. Existing studies [1, 2] have already conducted more comprehensive comparisons across different RAG and GraphRAG approaches. The authors should include more representative retrieval methods rather than drawing broad conclusions from a single graph-based baseline.

  3. Limited dataset diversity and generalizability. The experiments are restricted to three traditional multi-hop QA datasets: MuSiQue, 2WikiMultiHopQA, and HotpotQA. Moreover, several important conclusions are primarily supported by the MuSiQue four-hop subset, which contains only 166 questions. The results vary considerably across datasets, raising concerns about their generalizability. A broader evaluation covering more challenging and diverse retrieval tasks is necessary.

  4. Potential data contamination from LLM pretraining. All three evaluation datasets have been publicly available for years and may have appeared in the pretraining or post-training data of the evaluated LLMs. This is particularly concerning given the use of recent frontier models.

  5. Unfair comparison of computational costs. The paper matches the number of retrieved passages between different approaches, but iterative retrieval requires multiple LLM calls and additional reasoning. Matching passage counts alone does not establish a fair comparison. A more comprehensive evaluation of computational cost, latency, and retrieval efficiency is needed.

[1] Han et al. RAG vs. GraphRAG: A Systematic Evaluation and Key Insights.

[2] Xiang et al. When to Use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation

[3] Liu et al. Lost in the Middle: How Language Models Use Long Contexts.

Questions for authors

Please refer to the weaknesses.

Reviewer_PbubOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a diagnostic analysis that decomposes the errors of RAG systems into retrieval errors and answer-generation errors.

Critical strengths
  1. The experiments are well controlled and cover multiple models.
  2. The evaluation and analysis are detailed and help explain where errors come from on multi-hop questions.
Critical weaknesses
  1. The novelty and significance appear limited.
  • The key findings closely resemble those of existing work (Joren et al., ICLR 2025; Jin et al., ICLR 2025).
  • The proposed solution relies on a standard distillation method, and the training result (search transfers while reading does not) seems expected given prior work on search-only training, such as s3 (EMNLP 2025; https://arxiv.org/abs/2505.14146).
  1. The evaluation covers only multi-hop questions and does not show the effect on other question types, so it is unclear whether the conclusions generalize.
  2. The claim that "accuracy misranks retrieval" appears to be contradicted by the paper's own token-matched results.
Questions for authors
  1. Beyond the tighter methodology, what conclusions does this paper reach that the previous works could not?
  2. The abstract states that "training moves one share and not the other." This may be stronger than the evidence supports: the paper trains only the retriever, not the reader, so it does not show that training the system end to end would leave the reader unchanged. Could the authors clarify or qualify this claim?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.