- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents LI-RAG, a multi-round dense retrieval framework that distills LLM-generated question decomposition traces into a query encoder. The encoder is trained with a sub-question loss and a memory retrieval loss, allowing it to update retrieval queries based on previously retrieved passages without intermediate LLM calls. Experiments across multi-hop QA benchmarks demonstrate improved retrieval and answer accuracy with reduced online LLM usage.
Critical strengths
- Distilling LLM-generated sub-questions into latent query embeddings is an interesting and intuitive approach to multi-hop retrieval. The combination of sub-question supervision and retrieval objectives is well motivated.
- Avoiding intermediate LLM calls also offers a practical efficiency advantage.
Critical weaknesses
- The retrieval process is not fully adaptive. The main experiments use a fixed three-round retrieval procedure, regardless of whether sufficient evidence has already been retrieved. Although the appendix explores a retrieval-overlap-based stopping heuristic, evidence-aware stopping remains underexplored, leaving room for improving the effectiveness-efficiency trade-off.
- The evaluation mainly relies on relatively small retrieval corpora. It remains unclear whether the improvements would generalize to larger, more realistic document collections, where retrieving relevant evidence and maintaining useful multi-round context may be more challenging.
Questions for authors
- Can the authors further investigate adaptive stopping based on evidence sufficiency? In particular, how would it affect retrieval accuracy, answer quality, and inference cost for questions requiring different numbers of reasoning hops?
- How well does LI-RAG scale to substantially larger retrieval corpora? An evaluation under more realistic retrieval settings would help establish the generality of the reported improvements.
6 comments
Response to Reviewer nCnr (2/3): W2/Q2 Part 1 - scalability with comparable baselines
W2 / Q2. Scaling to larger corpora
Setup. Using the HotpotQA 2017 Wikipedia dump, we grew the MuSiQue (11,656 passages), 2Wiki (6,119) and MultiHop-RAG (6,055) test corpora to 20k, 40k, 100k and 1M passages. The test questions and their gold passages are unchanged, and added passages whose title already appears in the corpus are excluded, so every added passage is a distractor. All methods use the paper's checkpoints and settings, and every reader receives nine passages. GRITHopper is the paper's version, retrained on the same 4,000 MuSiQue questions as LI-RAG.
Answer accuracy (%)
| Benchmark | Method | 20k | 40k | 100k | 1M |
|---|---|---|---|---|---|
| MuSiQue | LI-RAG | 58.2 | 56.4 | 54.7 | 49.9 |
| GRITHopper | |||||
| HippoRAG 2 | 49.9 | 49.3 | 45.5 | – | |
| TreeHop | 46.9 | 44.2 | 41.8 | 33.2 | |
| Single-round dense | 46.1 | 43.2 | 41.8 | 31.3 | |
| 2Wiki | LI-RAG | 83.8 | 82.4 | 83.2 | 82.7 |
| GRITHopper | 66.8 | 68.6 | |||
| HippoRAG 2 | 67.1 | – | |||
| TreeHop | 56.9 | 56.5 | 57.6 | 57.8 | |
| Single-round dense | 57.2 | 56.9 | 57.5 | 56.7 | |
| MultiHop-RAG | LI-RAG | 77.6 | 77.8 | 77.3 | 76.7 |
| GRITHopper | 69.8 | 69.5 | 68.9 | 68.8 | |
| HippoRAG 2 | 72.4 | 72.2 | 72.1 | – | |
| TreeHop | 73.0 | 72.7 | |||
| Single-round dense | 72.6 | 72.3 |
HippoRAG 2 was not run at 1M: indexing 1M passages would take about 2M LLM calls, which is beyond our computation budget.
R@10 / FCR@10
| Benchmark | Method | 20k | 40k | 100k | 1M |
|---|---|---|---|---|---|
| MuSiQue | LI-RAG | 0.892 / 0.724 | 0.882 / 0.705 | 0.871 / 0.686 | 0.791 / 0.567 |
| GRITHopper | / | / | / | / | |
| HippoRAG 2 | 0.819 / 0.584 | 0.803 / 0.558 | 0.776 / 0.514 | – | |
| TreeHop | 0.772 / 0.494 | 0.757 / 0.469 | 0.735 / 0.432 | 0.640 / 0.305 | |
| Single-round dense | 0.772 / 0.493 | 0.758 / 0.473 | 0.734 / 0.433 | 0.633 / 0.292 | |
| 2Wiki | LI-RAG | 0.989 / 0.967 | 0.987 / 0.963 | 0.987 / 0.962 | 0.983 / 0.950 |
| GRITHopper | 0.915 / 0.727 | 0.911 / 0.719 | / 0.709 | / | |
| HippoRAG 2 | / | / | 0.888 / | – | |
| TreeHop | 0.797 / 0.562 | 0.793 / 0.555 | 0.785 / 0.544 | 0.757 / 0.490 | |
| Single-round dense | 0.800 / 0.567 | 0.793 / 0.555 | 0.786 / 0.545 | 0.755 / 0.485 | |
| MultiHop-RAG | LI-RAG | 0.603 / 0.313 | 0.603 / 0.312 | 0.603 / 0.312 | 0.597 / 0.305 |
| GRITHopper | 0.438 / 0.191 | 0.436 / 0.189 | 0.434 / 0.189 | 0.420 / 0.175 | |
| HippoRAG 2 | 0.573 / 0.283 | 0.575 / 0.285 | 0.574 / 0.279 | – | |
| TreeHop | / | / | / | / 0.286 | |
| Single-round dense | 0.593 / | 0.593 / | 0.593 / | / |
The improvements hold on the larger corpora. On MuSiQue, LI-RAG's margins over these methods are largest at 1M: 18.6 points over single-round dense (12.1 at 20k) and 10.1 points over GRITHopper (8.1 at 20k). From 20k to 1M, dense retrieval loses 14.8 points, LI-RAG 8.3.
Response to Reviewer nCnr (3/3): W2/Q2 Part 2 - scalability with uncomparable baselines
Baselines that are not directly comparable. IRCoT calls an LLM at every retrieval step, where LI-RAG calls none. GRITHopper-7B* is the model released by the GRITHopper authors, trained with gold evidence on 184,070 questions and claims, 46 times LI-RAG's 4,000. We therefore report them separately, with LI-RAG for reference; GRITHopper-7B* was run on MuSiQue only.
Answer accuracy (%), not directly comparable
| Benchmark | Method | 20k | 40k | 100k | 1M |
|---|---|---|---|---|---|
| MuSiQue | LI-RAG | 58.2 | 54.7 | 49.9 | |
| 2Wiki | LI-RAG | 83.8 | 82.4 | 83.2 | 82.7 |
| MultiHop-RAG | LI-RAG | 77.6 | 77.8 | 77.3 | 76.7 |
R@10 / FCR@10, not directly comparable
| Benchmark | Method | 20k | 40k | 100k | 1M |
|---|---|---|---|---|---|
| MuSiQue | LI-RAG | / | / | 0.871 / | 0.791 / 0.567 |
| 2Wiki | LI-RAG | 0.989 / 0.967 | 0.987 / 0.963 | 0.987 / 0.962 | 0.983 / 0.950 |
| MultiHop-RAG | LI-RAG | 0.603 / 0.313 | 0.603 / 0.312 | 0.603 / 0.312 | 0.597 / 0.305 |
LI-RAG is within 2 points of IRCoT and GRITHopper-7B* at every size on MuSiQue and MultiHop-RAG, and within 2.6 points of IRCoT on 2Wiki. On MuSiQue it has the highest R@10 and FCR@10 at 1M.
We have added these results to Appendix G of the revised paper (Tables 9–13), and will revise the limitation section accordingly. Reviewer 1L2p also raised the large-corpus setting and asked about latency; the retrieval latency at 100k and 1M passages is in Table 13.
Response to Reviewer nCnr (1/3)
We thank the reviewer for the review. We address each point below.
W1 / Q1. Adaptive stopping based on evidence sufficiency
In the paper, we have explored a simple training-free rule for adaptive stopping, the overlap stop. Following the reviewer's suggestion, we also tried a stop based on evidence sufficiency: a stop head trained to predict whether the passages retrieved so far contain all the evidence needed. The overlap stop works better. A simple stop is therefore already effective for LI-RAG, and more advanced stopping methods need further research.
Stop head. We trained a stop head, a single linear layer on the encoder's pooled output, jointly with the paper's two losses. Its input is the input of the next round, the question and the passages retrieved so far, so it reads the same forward pass that produces the next query. Its output is the probability that these passages already contain all the evidence needed to answer. It is trained with binary cross-entropy against a label that is 1 when the passages in its input include all of the question's gold supporting passages, and 0 otherwise. At inference, the loop stops once this probability reaches 0.5, with a cap of five rounds.
For a fair comparison, we compare it with the overlap stop of Appendix D on the MuSiQue + HotpotQA model: both models are trained on the same 7,000 MuSiQue and HotpotQA questions with the same settings, and the head model only adds the head and its loss. As in Appendix D, the reader receives the retrieved passages.
Results. Answer accuracy (%) / rounds per question on the test sets:
On MuSiQue, by number of hops:
On 2Wiki, by number of hops (comparison, compositional and inference questions have two gold passages, bridge-comparison questions four):
The overlap stop, which needs no training, already performs well: it keeps 1.4 to 1.6 rounds on the single-hop NQ and PopQA and 1.9 to 2.7 on the multi-hop benchmarks. The learned head does not improve on it, keeping more rounds on four of the five benchmarks and losing 5.0 points on 2Wiki, on the 4-hop questions. A stop based on evidence sufficiency needs more careful design. We will add these results to Appendix D.