acceptodds
Under review as a conference paper at ICLR 2027

LI-RAG: Latent Iterative RAG for Complex QA

Abstract

Retrieval-augmented generation (RAG) grounds large language models (LLMs) in retrieved passages, but for multi-hop questions a single round of retrieval often misses the later evidence, which cannot be located from the question alone. Existing multi-round methods either call an LLM at every retrieval step, so that latency grows with the number of hops, or train the retriever on annotated evidence chains, which are expensive to collect. We propose LI-RAG (Latent Iterative RAG), which trains a multi-round query encoder on the decomposition reasoning paths of an LLM, supervising each round's query embedding with the sub-question the LLM asks at that step and the evidence it selects. At inference, retrieval takes only a few encoder forward passes and no LLM call. Trained on 4,000 questions, LI-RAG outperforms multi-round retrievers trained on the gold evidence of the same questions on four multi-hop benchmarks, performs on par with or better than a retriever trained on 46 times as much annotated data, and remains competitive with a method that calls an LLM at every retrieval step.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-hop QA, RAG, embedding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nCnrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LI-RAG, a multi-round dense retrieval framework that distills LLM-generated question decomposition traces into a query encoder. The encoder is trained with a sub-question loss and a memory retrieval loss, allowing it to update retrieval queries based on previously retrieved passages without intermediate LLM calls. Experiments across multi-hop QA benchmarks demonstrate improved retrieval and answer accuracy with reduced online LLM usage.

Critical strengths
  1. Distilling LLM-generated sub-questions into latent query embeddings is an interesting and intuitive approach to multi-hop retrieval. The combination of sub-question supervision and retrieval objectives is well motivated.
  2. Avoiding intermediate LLM calls also offers a practical efficiency advantage.
Critical weaknesses
  1. The retrieval process is not fully adaptive. The main experiments use a fixed three-round retrieval procedure, regardless of whether sufficient evidence has already been retrieved. Although the appendix explores a retrieval-overlap-based stopping heuristic, evidence-aware stopping remains underexplored, leaving room for improving the effectiveness-efficiency trade-off.
  2. The evaluation mainly relies on relatively small retrieval corpora. It remains unclear whether the improvements would generalize to larger, more realistic document collections, where retrieving relevant evidence and maintaining useful multi-round context may be more challenging.
Questions for authors
  1. Can the authors further investigate adaptive stopping based on evidence sufficiency? In particular, how would it affect retrieval accuracy, answer quality, and inference cost for questions requiring different numbers of reasoning hops?
  2. How well does LI-RAG scale to substantially larger retrieval corpora? An evaluation under more realistic retrieval settings would help establish the generality of the reported improvements.
6 comments
AuthorsOpenReview ↗

Response to Reviewer nCnr (1/3)

We thank the reviewer for the review. We address each point below.

W1 / Q1. Adaptive stopping based on evidence sufficiency

In the paper, we have explored a simple training-free rule for adaptive stopping, the overlap stop. Following the reviewer's suggestion, we also tried a stop based on evidence sufficiency: a stop head trained to predict whether the passages retrieved so far contain all the evidence needed. The overlap stop works better. A simple stop is therefore already effective for LI-RAG, and more advanced stopping methods need further research.

Stop head. We trained a stop head, a single linear layer on the encoder's pooled output, jointly with the paper's two losses. Its input is the input of the next round, the question and the passages retrieved so far, so it reads the same forward pass that produces the next query. Its output is the probability that these passages already contain all the evidence needed to answer. It is trained with binary cross-entropy against a label that is 1 when the passages in its input include all of the question's gold supporting passages, and 0 otherwise. At inference, the loop stops once this probability reaches 0.5, with a cap of five rounds.

For a fair comparison, we compare it with the overlap stop of Appendix D on the MuSiQue + HotpotQA model: both models are trained on the same 7,000 MuSiQue and HotpotQA questions with the same settings, and the head model only adds the head and its loss. As in Appendix D, the reader receives the retrieved passages.

Results. Answer accuracy (%) / rounds per question on the test sets:

MethodMuSiQueHotpotQA2WikiNQPopQA
Overlap stop (cap 5)56.9 / 2.785.6 / 1.979.7 / 2.470.6 / 1.472.6 / 1.6
Stop head (cap 5)56.7 / 3.386.3 / 2.074.7 / 2.470.4 / 1.870.9 / 1.9

On MuSiQue, by number of hops:

MethodRounds2-hop3-hop4-hopAll
Overlap stop (cap 5)2.766.456.328.356.9
Stop head (cap 5)3.364.957.629.556.7

On 2Wiki, by number of hops (comparison, compositional and inference questions have two gold passages, bridge-comparison questions four):

MethodRounds2-hop4-hopAll
Overlap stop (cap 5)2.480.876.279.7
Stop head (cap 5)2.481.751.974.7

The overlap stop, which needs no training, already performs well: it keeps 1.4 to 1.6 rounds on the single-hop NQ and PopQA and 1.9 to 2.7 on the multi-hop benchmarks. The learned head does not improve on it, keeping more rounds on four of the five benchmarks and losing 5.0 points on 2Wiki, on the 4-hop questions. A stop based on evidence sufficiency needs more careful design. We will add these results to Appendix D.

AuthorsOpenReview ↗

Response to Reviewer nCnr (2/3): W2/Q2 Part 1 - scalability with comparable baselines

W2 / Q2. Scaling to larger corpora

Setup. Using the HotpotQA 2017 Wikipedia dump, we grew the MuSiQue (11,656 passages), 2Wiki (6,119) and MultiHop-RAG (6,055) test corpora to 20k, 40k, 100k and 1M passages. The test questions and their gold passages are unchanged, and added passages whose title already appears in the corpus are excluded, so every added passage is a distractor. All methods use the paper's checkpoints and settings, and every reader receives nine passages. GRITHopper is the paper's version, retrained on the same 4,000 MuSiQue questions as LI-RAG.

Answer accuracy (%)

BenchmarkMethod20k40k100k1M
MuSiQueLI-RAG58.256.454.749.9
GRITHopper50.1‾\underline{\text{50.1}}49.7‾\underline{\text{49.7}}47.1‾\underline{\text{47.1}}39.8‾\underline{\text{39.8}}
HippoRAG 249.949.345.5–
TreeHop46.944.241.833.2
Single-round dense46.143.241.831.3
2WikiLI-RAG83.882.483.282.7
GRITHopper66.868.668.1‾\underline{\text{68.1}}66.9‾\underline{\text{66.9}}
HippoRAG 270.8‾\underline{\text{70.8}}70.5‾\underline{\text{70.5}}67.1–
TreeHop56.956.557.657.8
Single-round dense57.256.957.556.7
MultiHop-RAGLI-RAG77.677.877.376.7
GRITHopper69.869.568.968.8
HippoRAG 272.472.272.1–
TreeHop73.073.3‾\underline{\text{73.3}}72.772.8‾\underline{\text{72.8}}
Single-round dense73.1‾\underline{\text{73.1}}72.673.3‾\underline{\text{73.3}}72.3

HippoRAG 2 was not run at 1M: indexing 1M passages would take about 2M LLM calls, which is beyond our computation budget.

R@10 / FCR@10

BenchmarkMethod20k40k100k1M
MuSiQueLI-RAG0.892 / 0.7240.882 / 0.7050.871 / 0.6860.791 / 0.567
GRITHopper0.836‾\underline{\text{0.836}} / 0.653‾\underline{\text{0.653}}0.815‾\underline{\text{0.815}} / 0.622‾\underline{\text{0.622}}0.790‾\underline{\text{0.790}} / 0.573‾\underline{\text{0.573}}0.676‾\underline{\text{0.676}} / 0.416‾\underline{\text{0.416}}
HippoRAG 20.819 / 0.5840.803 / 0.5580.776 / 0.514–
TreeHop0.772 / 0.4940.757 / 0.4690.735 / 0.4320.640 / 0.305
Single-round dense0.772 / 0.4930.758 / 0.4730.734 / 0.4330.633 / 0.292
2WikiLI-RAG0.989 / 0.9670.987 / 0.9630.987 / 0.9620.983 / 0.950
GRITHopper0.915 / 0.7270.911 / 0.7190.906‾\underline{\text{0.906}} / 0.7090.875‾\underline{\text{0.875}} / 0.661‾\underline{\text{0.661}}
HippoRAG 20.929‾\underline{\text{0.929}} / 0.806‾\underline{\text{0.806}}0.915‾\underline{\text{0.915}} / 0.770‾\underline{\text{0.770}}0.888 / 0.712‾\underline{\text{0.712}}–
TreeHop0.797 / 0.5620.793 / 0.5550.785 / 0.5440.757 / 0.490
Single-round dense0.800 / 0.5670.793 / 0.5550.786 / 0.5450.755 / 0.485
MultiHop-RAGLI-RAG0.603 / 0.3130.603 / 0.3120.603 / 0.3120.597 / 0.305
GRITHopper0.438 / 0.1910.436 / 0.1890.434 / 0.1890.420 / 0.175
HippoRAG 20.573 / 0.2830.575 / 0.2850.574 / 0.279–
TreeHop0.594‾\underline{\text{0.594}} / 0.288‾\underline{\text{0.288}}0.594‾\underline{\text{0.594}} / 0.288‾\underline{\text{0.288}}0.594‾\underline{\text{0.594}} / 0.288‾\underline{\text{0.288}}0.592‾\underline{\text{0.592}} / 0.286
Single-round dense0.593 / 0.288‾\underline{\text{0.288}}0.593 / 0.288‾\underline{\text{0.288}}0.593 / 0.288‾\underline{\text{0.288}}0.592‾\underline{\text{0.592}} / 0.287‾\underline{\text{0.287}}

The improvements hold on the larger corpora. On MuSiQue, LI-RAG's margins over these methods are largest at 1M: 18.6 points over single-round dense (12.1 at 20k) and 10.1 points over GRITHopper (8.1 at 20k). From 20k to 1M, dense retrieval loses 14.8 points, LI-RAG 8.3.

AuthorsOpenReview ↗

Response to Reviewer nCnr (3/3): W2/Q2 Part 2 - scalability with uncomparable baselines

Baselines that are not directly comparable. IRCoT calls an LLM at every retrieval step, where LI-RAG calls none. GRITHopper-7B* is the model released by the GRITHopper authors, trained with gold evidence on 184,070 questions and claims, 46 times LI-RAG's 4,000. We therefore report them separately, with LI-RAG for reference; GRITHopper-7B* was run on MuSiQue only.

Answer accuracy (%), not directly comparable

BenchmarkMethod20k40k100k1M
MuSiQueLI-RAG58.256.4‾\underline{\text{56.4}}54.749.9
IRCoT\color{gray}{\text{IRCoT}}58.0‾\color{gray}{\underline{\text{58.0}}}57.3\color{gray}{\textbf{57.3}}54.8‾\color{gray}{\underline{\text{54.8}}}49.8‾\color{gray}{\underline{\text{49.8}}}
GRITHopper-7B*\color{gray}{\text{GRITHopper-7B*}}56.7\color{gray}{\text{56.7}}56.1\color{gray}{\text{56.1}}56.4\color{gray}{\textbf{56.4}}48.9\color{gray}{\text{48.9}}
2WikiLI-RAG83.882.483.282.7
IRCoT\color{gray}{\text{IRCoT}}85.6\color{gray}{\textbf{85.6}}85.0\color{gray}{\textbf{85.0}}84.6\color{gray}{\textbf{84.6}}85.0\color{gray}{\textbf{85.0}}
MultiHop-RAGLI-RAG77.677.877.376.7
IRCoT\color{gray}{\text{IRCoT}}75.7\color{gray}{\text{75.7}}76.0\color{gray}{\text{76.0}}76.3\color{gray}{\text{76.3}}76.0\color{gray}{\text{76.0}}

R@10 / FCR@10, not directly comparable

BenchmarkMethod20k40k100k1M
MuSiQueLI-RAG0.892‾\underline{\text{0.892}} / 0.724‾\underline{\text{0.724}}0.882‾\underline{\text{0.882}} / 0.705‾\underline{\text{0.705}}0.871 / 0.686‾\underline{\text{0.686}}0.791 / 0.567
IRCoT\color{gray}{\text{IRCoT}}0.899 / 0.757\color{gray}{\textbf{0.899}\text{ / }\textbf{0.757}}0.892 / 0.739\color{gray}{\textbf{0.892}\text{ / }\textbf{0.739}}0.870‾ / 0.697\color{gray}{\underline{\text{0.870}}\text{ / }\textbf{0.697}}0.771‾ / 0.541‾\color{gray}{\underline{\text{0.771}}\text{ / }\underline{\text{0.541}}}
GRITHopper-7B*\color{gray}{\text{GRITHopper-7B*}}0.882 / 0.716\color{gray}{\text{0.882 / 0.716}}0.876 / 0.701\color{gray}{\text{0.876 / 0.701}}0.861 / 0.676\color{gray}{\text{0.861 / 0.676}}0.769 / 0.536\color{gray}{\text{0.769 / 0.536}}
2WikiLI-RAG0.989 / 0.9670.987 / 0.9630.987 / 0.9620.983 / 0.950
IRCoT\color{gray}{\text{IRCoT}}0.995 / 0.988\color{gray}{\textbf{0.995}\text{ / }\textbf{0.988}}0.994 / 0.985\color{gray}{\textbf{0.994}\text{ / }\textbf{0.985}}0.993 / 0.985\color{gray}{\textbf{0.993}\text{ / }\textbf{0.985}}0.990 / 0.978\color{gray}{\textbf{0.990}\text{ / }\textbf{0.978}}
MultiHop-RAGLI-RAG0.603 / 0.3130.603 / 0.3120.603 / 0.3120.597 / 0.305
IRCoT\color{gray}{\text{IRCoT}}0.613 / 0.322\color{gray}{\textbf{0.613}\text{ / }\textbf{0.322}}0.615 / 0.323\color{gray}{\textbf{0.615}\text{ / }\textbf{0.323}}0.621 / 0.323\color{gray}{\textbf{0.621}\text{ / }\textbf{0.323}}0.607 / 0.310\color{gray}{\textbf{0.607}\text{ / }\textbf{0.310}}

LI-RAG is within 2 points of IRCoT and GRITHopper-7B* at every size on MuSiQue and MultiHop-RAG, and within 2.6 points of IRCoT on 2Wiki. On MuSiQue it has the highest R@10 and FCR@10 at 1M.

We have added these results to Appendix G of the revised paper (Tables 9–13), and will revise the limitation section accordingly. Reviewer 1L2p also raised the large-corpus setting and asked about latency; the retrieval latency at 100k and 1M passages is in Table 13.

Reviewer_1L2pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LI-RAG tackles two problems in complex multi-hop QA: single-round retrieval rarely gathers all evidence, and calling large models round by round is costly. It uses sub-questions and evidence-selection trajectories generated offline by large models, and trains the query encoder with two objectives, sub-question vector alignment and evidence retrieval, to seek the next step of information from existing evidence. At inference, only a few rounds of encoder computation are needed, with no large model invoked during retrieval; the large model is called only at the end to generate the answer. Trained on trajectories from just 4,000 questions, it matches or exceeds comparable-cost baselines on four multi-hop QA benchmarks and approaches round-by-round large-model methods, showing strong data efficiency and effectiveness.

Critical strengths

The paper jointly uses sub-questions and evidence-selection trajectories generated by large models to train the query encoder, so that multi-round retrieval does not require online calls to large models. The design is clear and practically valuable. With training trajectories from only 4,000 questions, it achieves competitive results on multiple multi-hop QA benchmarks, demonstrating good data efficiency and cross-dataset generalization.

Critical weaknesses

The experimental corpus contains at most only about 12,000 passages, which is not yet sufficient to demonstrate the method's retrieval effectiveness and efficiency advantages on large-scale corpora. The experiments use only a single random seed and adopt large-model judging of answers, and some performance differences fall within the roughly 2-percentage-point fluctuation range reported by the authors, so the reliability of small improvements still needs verification.

Questions for authors
  • Could you add comparisons of retrieval effectiveness, end-to-end latency, and computational cost on a corpus with millions of passages to verify the method's scalability?
  • Could you add human evaluation of a subset of answers to confirm whether the small performance improvements are stable and reliable?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1L2p (1/3): W1/Q1 Part 1 - scalability with comparable baselines

We thank the reviewer for the review. We address each point below.

W1 / Q1. Effectiveness, latency and cost on a million-passage corpus

Setup. Using the HotpotQA 2017 Wikipedia dump, we grew the MuSiQue (11,656 passages), 2Wiki (6,119) and MultiHop-RAG (6,055) test corpora to 20k, 40k, 100k and 1M passages. The test questions and their gold passages are unchanged, and added passages whose title already appears in the corpus are excluded, so every added passage is a distractor. All methods use the paper's checkpoints and settings, and every reader receives nine passages. GRITHopper is the paper's version, retrained on the same 4,000 MuSiQue questions as LI-RAG.

Answer accuracy (%)

BenchmarkMethod20k40k100k1M
MuSiQueLI-RAG58.256.454.749.9
GRITHopper50.1‾\underline{\text{50.1}}49.7‾\underline{\text{49.7}}47.1‾\underline{\text{47.1}}39.8‾\underline{\text{39.8}}
HippoRAG 249.949.345.5–
TreeHop46.944.241.833.2
Single-round dense46.143.241.831.3
2WikiLI-RAG83.882.483.282.7
GRITHopper66.868.668.1‾\underline{\text{68.1}}66.9‾\underline{\text{66.9}}
HippoRAG 270.8‾\underline{\text{70.8}}70.5‾\underline{\text{70.5}}67.1–
TreeHop56.956.557.657.8
Single-round dense57.256.957.556.7
MultiHop-RAGLI-RAG77.677.877.376.7
GRITHopper69.869.568.968.8
HippoRAG 272.472.272.1–
TreeHop73.073.3‾\underline{\text{73.3}}72.772.8‾\underline{\text{72.8}}
Single-round dense73.1‾\underline{\text{73.1}}72.673.3‾\underline{\text{73.3}}72.3

HippoRAG 2 was not run at 1M: indexing 1M passages would take about 2M LLM calls, which is beyond our computation budget.

R@10 / FCR@10

BenchmarkMethod20k40k100k1M
MuSiQueLI-RAG0.892 / 0.7240.882 / 0.7050.871 / 0.6860.791 / 0.567
GRITHopper0.836‾\underline{\text{0.836}} / 0.653‾\underline{\text{0.653}}0.815‾\underline{\text{0.815}} / 0.622‾\underline{\text{0.622}}0.790‾\underline{\text{0.790}} / 0.573‾\underline{\text{0.573}}0.676‾\underline{\text{0.676}} / 0.416‾\underline{\text{0.416}}
HippoRAG 20.819 / 0.5840.803 / 0.5580.776 / 0.514–
TreeHop0.772 / 0.4940.757 / 0.4690.735 / 0.4320.640 / 0.305
Single-round dense0.772 / 0.4930.758 / 0.4730.734 / 0.4330.633 / 0.292
2WikiLI-RAG0.989 / 0.9670.987 / 0.9630.987 / 0.9620.983 / 0.950
GRITHopper0.915 / 0.7270.911 / 0.7190.906‾\underline{\text{0.906}} / 0.7090.875‾\underline{\text{0.875}} / 0.661‾\underline{\text{0.661}}
HippoRAG 20.929‾\underline{\text{0.929}} / 0.806‾\underline{\text{0.806}}0.915‾\underline{\text{0.915}} / 0.770‾\underline{\text{0.770}}0.888 / 0.712‾\underline{\text{0.712}}–
TreeHop0.797 / 0.5620.793 / 0.5550.785 / 0.5440.757 / 0.490
Single-round dense0.800 / 0.5670.793 / 0.5550.786 / 0.5450.755 / 0.485
MultiHop-RAGLI-RAG0.603 / 0.3130.603 / 0.3120.603 / 0.3120.597 / 0.305
GRITHopper0.438 / 0.1910.436 / 0.1890.434 / 0.1890.420 / 0.175
HippoRAG 20.573 / 0.2830.575 / 0.2850.574 / 0.279–
TreeHop0.594‾\underline{\text{0.594}} / 0.288‾\underline{\text{0.288}}0.594‾\underline{\text{0.594}} / 0.288‾\underline{\text{0.288}}0.594‾\underline{\text{0.594}} / 0.288‾\underline{\text{0.288}}0.592‾\underline{\text{0.592}} / 0.286
Single-round dense0.593 / 0.288‾\underline{\text{0.288}}0.593 / 0.288‾\underline{\text{0.288}}0.593 / 0.288‾\underline{\text{0.288}}0.592‾\underline{\text{0.592}} / 0.287‾\underline{\text{0.287}}

On MuSiQue, LI-RAG's margins over these methods are largest at 1M: 18.6 points over single-round dense (12.1 at 20k) and 10.1 points over GRITHopper (8.1 at 20k).

AuthorsOpenReview ↗

Response to Reviewer 1L2p (2/3): W1/Q1 Part 2 - scalability with uncomparable baselines

Baselines that are not directly comparable. IRCoT calls an LLM at every retrieval step, where LI-RAG calls none. GRITHopper-7B* is the model released by the GRITHopper authors, trained with gold evidence on 184,070 questions and claims, 46 times LI-RAG's 4,000. We therefore report them separately, with LI-RAG for reference; GRITHopper-7B* was run on MuSiQue only.

Answer accuracy (%), not directly comparable

BenchmarkMethod20k40k100k1M
MuSiQueLI-RAG58.256.4‾\underline{\text{56.4}}54.749.9
IRCoT\color{gray}{\text{IRCoT}}58.0‾\color{gray}{\underline{\text{58.0}}}57.3\color{gray}{\textbf{57.3}}54.8‾\color{gray}{\underline{\text{54.8}}}49.8‾\color{gray}{\underline{\text{49.8}}}
GRITHopper-7B*\color{gray}{\text{GRITHopper-7B*}}56.7\color{gray}{\text{56.7}}56.1\color{gray}{\text{56.1}}56.4\color{gray}{\textbf{56.4}}48.9\color{gray}{\text{48.9}}
2WikiLI-RAG83.882.483.282.7
IRCoT\color{gray}{\text{IRCoT}}85.6\color{gray}{\textbf{85.6}}85.0\color{gray}{\textbf{85.0}}84.6\color{gray}{\textbf{84.6}}85.0\color{gray}{\textbf{85.0}}
MultiHop-RAGLI-RAG77.677.877.376.7
IRCoT\color{gray}{\text{IRCoT}}75.7\color{gray}{\text{75.7}}76.0\color{gray}{\text{76.0}}76.3\color{gray}{\text{76.3}}76.0\color{gray}{\text{76.0}}

R@10 / FCR@10, not directly comparable

BenchmarkMethod20k40k100k1M
MuSiQueLI-RAG0.892‾\underline{\text{0.892}} / 0.724‾\underline{\text{0.724}}0.882‾\underline{\text{0.882}} / 0.705‾\underline{\text{0.705}}0.871 / 0.686‾\underline{\text{0.686}}0.791 / 0.567
IRCoT\color{gray}{\text{IRCoT}}0.899 / 0.757\color{gray}{\textbf{0.899}\text{ / }\textbf{0.757}}0.892 / 0.739\color{gray}{\textbf{0.892}\text{ / }\textbf{0.739}}0.870‾ / 0.697\color{gray}{\underline{\text{0.870}}\text{ / }\textbf{0.697}}0.771‾ / 0.541‾\color{gray}{\underline{\text{0.771}}\text{ / }\underline{\text{0.541}}}
GRITHopper-7B*\color{gray}{\text{GRITHopper-7B*}}0.882 / 0.716\color{gray}{\text{0.882 / 0.716}}0.876 / 0.701\color{gray}{\text{0.876 / 0.701}}0.861 / 0.676\color{gray}{\text{0.861 / 0.676}}0.769 / 0.536\color{gray}{\text{0.769 / 0.536}}
2WikiLI-RAG0.989 / 0.9670.987 / 0.9630.987 / 0.9620.983 / 0.950
IRCoT\color{gray}{\text{IRCoT}}0.995 / 0.988\color{gray}{\textbf{0.995}\text{ / }\textbf{0.988}}0.994 / 0.985\color{gray}{\textbf{0.994}\text{ / }\textbf{0.985}}0.993 / 0.985\color{gray}{\textbf{0.993}\text{ / }\textbf{0.985}}0.990 / 0.978\color{gray}{\textbf{0.990}\text{ / }\textbf{0.978}}
MultiHop-RAGLI-RAG0.603 / 0.3130.603 / 0.3120.603 / 0.3120.597 / 0.305
IRCoT\color{gray}{\text{IRCoT}}0.613 / 0.322\color{gray}{\textbf{0.613}\text{ / }\textbf{0.322}}0.615 / 0.323\color{gray}{\textbf{0.615}\text{ / }\textbf{0.323}}0.621 / 0.323\color{gray}{\textbf{0.621}\text{ / }\textbf{0.323}}0.607 / 0.310\color{gray}{\textbf{0.607}\text{ / }\textbf{0.310}}

LI-RAG is within 2 points of IRCoT and GRITHopper-7B* at every size on MuSiQue and MultiHop-RAG, and within 2.6 points of IRCoT on 2Wiki. On MuSiQue it has the highest R@10 and FCR@10 at 1M.

Latency and computational cost at scale. End-to-end retrieval latency at batch size 1, from the question to the retrieved passages, including every LLM call made during retrieval (ms per question, mean over 300 questions). The H100 machine used for the paper is not accessible at the moment, so we measured all methods on the same machine with two NVIDIA RTX 6000 Ada GPUs (48 GB each):

Method100k1M
LI-RAG316367
IRCoT3,0163,261
  • The encoder passes do not depend on the corpus size. The growth is the exact vector search, which costs every method the same and which an approximate-nearest-neighbour index would reduce for all of them.
  • IRCoT makes 3.8 LLM calls and sends about 30,000 prompt tokens per question at every corpus size; LI-RAG makes none. At 1M, LI-RAG's retrieval is 8.9× faster than IRCoT's.

We have added these results to Appendix G of the revised paper (Tables 9–13), and will revise the limitation section accordingly.

AuthorsOpenReview ↗

Response to Reviewer 1L2p (3/3)

W2. Single seed and LLM judging

Our paper does not aim to outperform IRCoT or the released GRITHopper-7B*, which use far more resources: IRCoT makes 2.3 to 4.3 LLM calls per question during retrieval, where LI-RAG makes none (Section 4.2), and GRITHopper-7B* is trained with gold evidence on 184,070 questions and claims, 46 times LI-RAG's 4,000 (Table 4). Our claim is that LI-RAG reaches their accuracy, and sometimes exceeds it, at a fraction of that cost: it is within 1.2 points of IRCoT on five of the six benchmarks and 2.3 points below on 2Wiki (Section 4.2), and it is ahead of GRITHopper-7B* by 1.7 points on MuSiQue and 10.7 on 2Wiki and within about two points elsewhere (Table 3). Differences below about two points are not established; the paper treats them as ties (Section 4.1), and our claims do not rest on them. The claimed improvements are larger: +11.1 and +23.3 points over single-round dense retrieval on MuSiQue and 2Wiki, and +3.1 to +14.0 points over GRITHopper retrained on the same questions on the four multi-hop benchmarks (Table 1). The retrieval metrics involve no LLM judge and show the same ordering: FCR@10 0.740 vs. 0.673 for the next-best ranked method on MuSiQue, and 0.970 vs. 0.852 on 2Wiki (Table 2).

We also retrained LI-RAG from scratch on a different machine (two NVIDIA RTX 6000 Ada GPUs, 48 GB each, instead of the paper's H100), regenerating the teacher's traces for the same 4,000 MuSiQue questions and keeping the paper's settings. Answer accuracy (%) / R@10:

ModelMuSiQueHotpotQA2WikiMultiHop-RAGNQPopQA
LI-RAG (paper)58.5 / 0.89986.1 / 0.99482.6 / 0.99077.3 / 0.60472.1 / 0.97172.4 / 0.814
LI-RAG (retrained)59.0 / 0.89786.1 / 0.99384.1 / 0.99278.6 / 0.62471.8 / 0.96972.1 / 0.831

Accuracy changes by at most 1.5 points and R@10 by at most 0.02, and every improvement listed above holds for the retrained model.

Q2. Human evaluation

We selected 15 MuSiQue questions that LI-RAG answers correctly and GRITHopper, retrained on the same 4,000 questions, does not, and 15 that LI-RAG answers correctly and HippoRAG 2 does not. We checked by hand whether each round of retrieval finds the passage needed for the next hop of the question. Appendix F of the revised paper gives two of each in full, with the passages retrieved in each round and our assessment of each round; we checked the other 26 in the same way.

In all 30, LI-RAG's rounds follow the question's decomposition path, retrieving the passages for successive hops in order. For example, for "Who sent naval ships to the body of water that joins the Atlantic and the sea where the Rhine ends?" (Appendix F.2.2), which both baselines answer wrongly:

  • Round 1 retrieves the Rhine, which ends in the North Sea.
  • Round 2 retrieves the North Sea, which becomes the English Channel and connects to the Atlantic.
  • Round 3 retrieves the Roman Republic: under Caesar, an invasion fleet was assembled in the English Channel.

Both baselines fail at the last hop. GRITHopper also retrieves the North Sea and the English Channel, but its third hop retrieves naval passages from other wars (Crimean, Finnish, Franco-Prussian), and its reader answers "The French". HippoRAG 2 retrieves once, through Personalized PageRank (PPR) over its knowledge graph, and ranks the Roman Republic passage 12th, outside the nine passages its reader receives; its reader also answers "The French". LI-RAG's multi-round retrieval solves such questions: the English Channel is not named in the question, only in the passage of the previous hop, and each round of LI-RAG reads the passages retrieved so far to find the next one.

Reviewer_Sic5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LI-RAG addresses multi-hop question answering by learning an iterative retrieval policy that avoids generating natural-language subquestions at inference time. During offline supervision construction, an LLM planner generates subquestions, while a solver identifies useful retrieved passages and produces intermediate answers. The query encoder is then trained to map the original question and previously retrieved evidence to a retrieval vector, using both alignment with teacher subquestion embeddings and a passage retrieval objective. The passage encoder and document index remain fixed.

The paper evaluates the method on four multi-hop and two single-hop QA benchmarks. The main model, trained using traces derived from 4,000 MuSiQue questions, achieves substantial improvements over single-pass retrieval in both answer accuracy and evidence retrieval. The results support the feasibility of transferring useful aspects of an LLM’s iterative search behavior into a query encoder without generating explicit subquestions during retrieval.

Critical strengths

S1. A useful supervision design for a practically relevant problem

The paper addresses an important challenge in multi-hop QA: retrieving successive pieces of evidence while avoiding an autoregressive query-generation step at every retrieval round. Its contribution lies in combining teacher subquestion targets and useful-passage supervision to train an evidence-conditioned retrieval policy while preserving a fixed document index.

Relevant ingredients have precedents. For example, ConvDR distills representations of rewritten queries, while GRITHopper performs iterative retrieval conditioned on previously retrieved passages. Against this background, LI-RAG offers a useful training design and empirical evidence for its effectiveness. Its value does not depend on introducing an entirely new retrieval primitive.

S2. Improvements are supported by both answer quality and evidence retrieval

The gains over the single-pass NV-Embed-v2 baseline are substantial on several benchmarks. On MuSiQue, QA accuracy increases from 47.4% to 58.5%, while FCR@10 increases from 0.504 to 0.740. On 2Wiki, the corresponding improvements are from 59.3% to 82.6% and from 0.578 to 0.970 (Tables 1–2).

The evidence retrieval results are particularly valuable because they provide support beyond LLM-judged answer accuracy. Together, these results make a convincing case that the complete training recipe improves retrieval of the evidence needed for answering. Comparisons with the strongest iterative baselines are closer and vary across datasets, but uniformly achieving the best score is not necessary for the method to constitute a meaningful contribution.

S3. The evaluation addresses several plausible alternative explanations

The paper compares against both released GRITHopper models and a version retrained with a matched 4,000-question training budget. It also evaluates LI-RAG with GritLM as the retriever backbone (Tables 1 and 3). The latter provides useful evidence that the gains are not solely attributable to choosing NV-Embed-v2: in particular, the substantial advantage over released GRITHopper on 2Wiki persists with GritLM.

Evaluation across multiple QA datasets, including the news-based MultiHop-RAG benchmark, also provides useful breadth. The interpretation of these results as transfer across datasets would benefit from clarifying the model-selection procedure, as discussed below.

Critical weaknesses

W1. The experiments establish the effectiveness of the complete recipe more clearly than the contribution of subquestion alignment

A central motivation is that subquestion supervision provides a particularly suitable target for learning iterative retrieval. However, the experiments do not fully isolate its contribution.

Table 7 includes a subquestion-only variant, but no LI-RAG variant trained with the memory retrieval loss alone under otherwise matched conditions. Moreover, this ablation uses the 10,000-question mixture of three datasets rather than the main 4,000-question MuSiQue setup, and reports retrieval metrics rather than downstream QA results. The GRITHopper comparisons do not resolve this attribution issue because the objectives and other training conditions also differ. Consequently, the results support the combined training recipe more strongly than they establish that subquestion alignment specifically explains its advantage over alternative supervision strategies.

The related claim that the subquestion targets are computable from information available to the student also needs a more precise justification. A planner prompt restricting unsupported entities does not, by itself, establish that all information used by the planner is retained in the student’s input. Similarly, passage ranking does not necessarily require a query representation to encode every fact in a positive passage, so the presence of previously unseen facts alone does not establish that passage-based supervision is unsuitable.

These concerns limit the attribution of the gains and the strength of the proposed explanation; they do not invalidate the empirical effectiveness of the full method (Sec. 3.2–3.4, Table 7, and Appendix A).

W2. The practical efficiency advantage is not quantified

Eliminating autoregressive query generation during retrieval and preserving a fixed document index are useful properties. However, the paper does not report inference latency or compute measurements sufficient to establish practical cost superiority over competing approaches.

The main configuration still performs three rounds of encoding over accumulated textual context using a 7.85B-parameter backbone. The small number of trainable adapter parameters does not directly determine inference cost. Counts of generative calls or generated tokens therefore establish only part of the efficiency picture.

The supported conclusion is that LI-RAG removes autoregressive query generation from the retrieval loop. The magnitude of any resulting advantage in end-to-end latency or compute remains unquantified, and broader efficiency claims should be qualified accordingly (Sec. 3.3, 3.5, and 4.2; Appendix A).

Questions for authors

Q1. Which development data and criteria were used to select the reported models?

Section 4.1 refers to 500 in-domain development questions per dataset, while Appendix A identifies the second epoch as the main model’s selected checkpoint. Please clarify:

  • Which development dataset or datasets, and which metric or aggregation criterion, were used to select the main checkpoint and training configuration?
  • Was exactly the same checkpoint used across all six evaluation datasets?
  • Did development results from any target dataset influence checkpoint selection or configuration choices for that dataset?

Q2. What ensures that the teacher’s subquestion targets are supported by information available to the student?

Please clarify how the trace construction process handles the following situations:

  • When the solver selects no useful passage, can its intermediate answer still enter the planner’s history and influence later subquestions?
  • Can passage retention or input truncation remove information from the student’s input that remains available to the planner through previous intermediate answers?
  • Is the intended accessibility of the subquestion targets enforced by the construction procedure, or primarily encouraged through the planner prompt?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Sic5 (1/2)

We thank the reviewer for the careful review. We address each point below.

W1. Contribution of sub-question alignment

New matched ablation. The H100 machine used for the paper is not accessible at the moment, so we trained three models on a machine with two NVIDIA RTX 6000 Ada GPUs (48 GB each), on the same 4,000 MuSiQue training questions, with the teacher's traces regenerated using the paper's teacher, prompts and loop. The three differ only in the loss weights: both losses, the memory retrieval loss only, and the sub-question loss only. They share the same traces, batch, seed and checkpoint-selection rule, and each is evaluated for answer accuracy and retrieval on all six benchmarks. The retrained full-loss model therefore differs slightly from the submitted one (MuSiQue 59.0 vs. 58.5, 2Wiki 84.1 vs. 82.6).

Answer accuracy (%) / FCR@10:

LossMuSiQueHotpotQA2WikiMultiHop-RAGNQPopQAAverage
Both (full)59.0 / 0.74486.1 / 0.98884.1 / 0.97878.6 / 0.333‾\underline{\text{0.333}}71.8 / 0.90472.1‾\underline{\text{72.1}} / 0.66275.3 / 0.768
Memory retrieval only58.3‾\underline{\text{58.3}} / 0.741‾\underline{\text{0.741}}85.6‾\underline{\text{85.6}} / 0.99183.4‾\underline{\text{83.4}} / 0.963‾\underline{\text{0.963}}77.4 / 0.31271.4‾\underline{\text{71.4}} / 0.906‾\underline{\text{0.906}}72.5 / 0.60974.8‾\underline{\text{74.8}} / 0.754‾\underline{\text{0.754}}
Sub-question only56.6 / 0.72385.0 / 0.990‾\underline{\text{0.990}}81.0 / 0.93177.6‾\underline{\text{77.6}} / 0.33771.8 / 0.90872.5 / 0.632‾\underline{\text{0.632}}74.1 / 0.753

Averaged over the six benchmarks, the full objective improves both answer accuracy and retrieval over either single loss (accuracy 75.3 vs. 74.8 and 74.1; FCR@10 0.768 vs. 0.754 and 0.753), and each single loss falls short in a different place.

What the paper proposes. Our contribution is the training recipe as a whole: distilling an LLM's decomposition path into the query encoder through two losses, both taken from the teacher's traces. The sub-question loss targets the teacher's sub-question, and the positives of the memory retrieval loss are the passages the teacher retrieved with that sub-question and the solver marked useful, not gold evidence. The paper defines the training loss as their sum and states that the two play different roles (Section 3.4), and the first contribution lists both. The ablation shows that this combination, rather than either loss alone, performs best.

Justification of the sub-question target. Our argument is not that passage supervision cannot rank passages, but that a passage target contains facts absent from the encoder's inputs and can be fitted on the training questions by memorising the training passages, which transfers less well to other entities (Section 1). The ablation is consistent with this: passage supervision alone is strong on the Wikipedia multi-hop chains, while adding the sub-question loss improves retrieval on MultiHop-RAG and PopQA. The accessibility of the sub-question targets is addressed under Q2.

In the revision, we will add this ablation to the appendix.

W2. Efficiency

We measured end-to-end retrieval latency at batch size 1, from the question to the retrieved passages, including every LLM call made during retrieval, on 300 test questions per benchmark. All methods ran inference on the same machine as in W1 (two NVIDIA RTX 6000 Ada GPUs, 48 GB each) with the same software stack. Means in ms, vLLM prefix caching on as in the paper's runs:

MethodLLM calls in retrieval (MuSiQue / 2Wiki)MuSiQue2Wiki
LI-RAG0310331
IRCoT3.9 / 4.33,0272,909
  • LI-RAG's retrieval is 9.8× faster than IRCoT's on MuSiQue and 8.8× on 2Wiki (14.6× and 14.4× without prefix caching). At the 95th percentile, it takes 447 ms against 9,688 ms on MuSiQue.

Larger corpora. Using the HotpotQA 2017 Wikipedia dump, we grew the MuSiQue test corpus to 100k and 1M passages and measured latency with the same protocol (ms per question, mean over 300 questions):

Method100k1M
LI-RAG316367
IRCoT3,0163,261
  • At 1M, LI-RAG's retrieval is 8.9× faster than IRCoT's, which makes 3.8 LLM calls per question at every corpus size where LI-RAG makes none. LI-RAG's encoder passes do not depend on the corpus size; the growth is the exact vector search, which costs every method the same and which an approximate-nearest-neighbour index would reduce for all of them.

We have added this table to Appendix G of the revised paper (Table 13).

AuthorsOpenReview ↗

Response to Reviewer Sic5 (2/2)

Q1. Model selection

  • Data and criterion. The checkpoint is selected only on MuSiQue, by the answer accuracy of the inference loop on its 500 development questions; recall is not used. We select on MuSiQue alone so that the other five benchmarks measure out-of-distribution performance.
  • Same checkpoint. Yes, within each model: every trained model is evaluated on all six benchmarks with one checkpoint.
    • Tables 1–3 and 5 compare LI-RAG with the baselines. They use the main model, trained on the traces of 4,000 MuSiQue questions, at epoch 2, selected on MuSiQue's development questions. The GritLM-7B version in Table 3 is a separate model, selected in the same way (epoch 3).
    • Tables 6–8 report the training ablations and early stopping. Each variant there is a separate model, whose checkpoint is selected on the development questions of the datasets it is trained on (e.g., MuSiQue and HotpotQA for the MuSiQue + HotpotQA model).
  • Influence of target datasets on checkpoint selection. None.

We agree that "500 in-domain development questions per dataset" in Section 4.1 was confusing. It does not mean that a different checkpoint is selected for each evaluation benchmark. For the main model, trained on traces of MuSiQue training questions, the checkpoint is selected by its answer accuracy on 500 MuSiQue development questions. For the MuSiQue + HotpotQA model, trained on traces of MuSiQue and HotpotQA training questions, it is selected by its mean answer accuracy on 500 MuSiQue and 500 HotpotQA development questions. The development questions come from each dataset's official development split, are disjoint from the test questions, and are not used for training. We have reworded this sentence in the revised paper (Section 4.1, lines 322–341).

Q2. Accessibility of the sub-question targets

We measured each case on the traces of the 4,000 MuSiQue training questions used for the retrained models in W1 (10,863 rounds):

  • No passage selected. The sub-answer does enter the planner's history, but the solver is instructed to answer "unknown" in this case and does so in 98.6% of such rounds. Only 19 of 6,863 later sub-questions (0.3%) contain a sub-answer from a round without a selected passage.
  • Retention and truncation. The student's input always includes the passages the solver selected in each earlier round, up to three per round. Only 3.0% of rounds select more than three, and at most 0.9% of inputs exceed the 2,048-token limit. In 94.8% of the later sub-questions that name an earlier sub-answer (4,830 of 5,094), that sub-answer also appears in the passages the student reads.
  • Construction or prompt. Both, without an explicit filter. By construction, the planner never sees passages, and the passages that support each sub-answer are always placed in the student's input. The prompts then restrict the planner to entities from the question or earlier sub-answers, and the solver to "unknown" when no passage answers.

We will describe this in Section 3.2 and add these statistics to the appendix. If we have misunderstood any of these questions, please let us know and we will clarify.

Dear Reviewer Sic5,

Thank you again for your careful review. We would be grateful to know whether these responses address your concerns, and we would be glad to clarify anything that remains.

Comments on the paper

1 comment

We thank the reviewers for their careful reviews. We summarize the main concerns and our responses below; changes to the paper are set in blue.

  • Larger corpora (1L2p W1/Q1, nCnr W2/Q2). We grew the MuSiQue, 2Wiki and MultiHop-RAG test corpora to 1M passages. LI-RAG stays best among the comparable methods at every size, with its largest MuSiQue margins at 1M (Appendix G, Tables 9–12).

  • Latency (Sic5 W2, 1L2p Q1). LI-RAG's end-to-end retrieval is 9.8× faster than IRCoT's on MuSiQue (response to Sic5, W2) and 8.9× faster at 1M passages (Appendix G, Table 13).

  • Sub-question loss (Sic5 W1). In a matched ablation on the main 4,000 MuSiQue questions, the full objective outperforms either loss alone on average over the six benchmarks (response to Sic5, W1).

  • Single seed and LLM judging (1L2p W2). Our claimed gains exceed two points, and the retrieval metrics, which involve no LLM judge, agree (Tables 1–2). We also retrained LI-RAG from scratch, and its performance is consistent with the paper's (response to 1L2p, W2).

  • Human evaluation (1L2p Q2). In 30 hand-checked MuSiQue questions, LI-RAG's rounds retrieve the passages of successive hops in order (Appendix F).

  • Adaptive stopping (nCnr W1/Q1). A trained stop head does not improve on the training-free overlap stop of Appendix D, and more advanced adaptive stopping needs further research (response to nCnr, W1/Q1).

  • Model selection (Sic5 Q1). Each model uses one checkpoint for all six benchmarks, selected on MuSiQue development questions only (revised Section 4.1).

  • Sub-question targets (Sic5 Q2). Trace statistics show that the planner's sub-questions rarely rely on information missing from the student's input (response to Sic5, Q2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.