- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces HEMLOCK, a graph-based retrieval pipeline for multi-hop question answering (QA). The pipeline has three stages: (1) initial segment retrieval using ColBERTv2, (2) construction of an entity-based graph from the retrieved segments, and (3) importance scoring of graph nodes via Personalized PageRank (PPR), followed by a final reranking step using a lightweight cross-encoder. The authors evaluate HEMLOCK on HotpotQA, 2WikiMultiHopQA, and MuSiQue, reporting improved retrieval recall compared to a series of retrieval systems, alongside gains in downstream QA performance compared to IRCoT and HippoRAG2.
Critical strengths
- The approach is computationally lightweight compared to current LLM-based retrieval methods. By using PPR and a small cross-encoder (22M parameters) instead of expensive LLM reasoning loops, the paper presents a more scalable alternative for context expansion.
Critical weaknesses
-
The core assumption of HEMLOCK is that all necessary evidence for a multi-hop query can be retrieved in a single pass, which is against the nature of multi-hop QA. Multi-hop QA is inherently sequential, as the answer to the n-th hop depends on the entity extracted from the (n-1)-th hop. These conditional dependencies are ignored by the present algorithm, as the PPR expands and explores the search space in one shot. For only few hops, or when hops have high entity overlap or relations of future hops are distinct, this can work. However, the approach is not tested on longer reasoning chains, with experiments in the paper being restricted to 2-hop questions.
-
The reported QA metrics are well below state-of-the-art results on 2WikiMultiHopQA and MuSiQue. For example, Interact-RAG (Hui et al., ICLR 2026) achieved 69.6 exact match (EM) on 2WikiMultiHopQA and 34.8 EM on MuSiQue with Qwen3-8B, and IGPO (Wang et al., ICLR 2026) 72.1 EM on 2WikiMultiHopQA and 32.7 EM on MuSiQue with Qwen2.5-7B, compared to the 29.3 EM and 20.8 EM achieved in this work. Only on HotpotQA there is an improvement compared to these approaches: HEMLOCK achieves 65.0 EM, while IGPO reports 59.0 EM and Interact-RAG an exact match of 51.6. While I recognize that the retrieval corpus and setup might differ, such a large gap requires discussion and comparisons to the state-of-the-art in comparable setups.
Questions for authors
-
How does the performance scale for questions as the number of hops increases?
-
The claim that HEMLOCK has no costs (e.g., "Multi-hop retrieval at $0 per query") is overstated. While HEMLOCK avoids expensive LLM calls, there is still an overhead added to the plain ColBERT retrieval: graph construction (NER and coreference resolution), PPR and a reranker that processes segments independently. Could the authors provide a more realistic measure for cost?
-
How does the work relate to prior work on graph-based QA, such as GRAFT-Net (Sun et al., EMNLP 2018) and PullNet (Sun et al., EMNLP 2019), which also employ PPR for exploring the graph?
-
Could the method benefit from stronger first-stage retrieval (e.g., RRF+CE), providing better seeds for the PPR walks?
Thank you for the review and for the suggestion about stronger seeds, which turned out to matter a great deal.
Clarification. The graph is built offline over the whole corpus, not from the retrieved segments. At query time it is only scored.
Sequential multi-hop. We have three results. First, hop strata: on MuSiQue 3/4-hop questions, HEMLOCK as submitted loses 4.9 points, whereas with stronger seeds the change is +0.5 (not significant) and 4-hop questions gain +2.0 (significant). Second, we implemented the sequential variant you describe, in which a second round is seeded from the first round's top evidence. We tried both ways of merging the rounds, re-applying the candidate cut or keeping everything, and neither changes recall by more than 0.7 points on any set. One-shot expansion is therefore not the bottleneck. The candidate cut is, and G3 fixes it. Third, breadth: on 2Wiki's 4-document questions, which have two parallel bridges, the gain is +11.3 as submitted and +15.0 with stronger seeds.
EM compared with the state of the art. The cited systems are multi-step LLM agents. HEMLOCK is a retriever that makes no LLM calls, so it corresponds to their search tool rather than to the whole agent. Interact-RAG's ablation shows that the gap comes from the agent. Its training-free workflow with plain search reaches 56.3 EM on 2Wiki, against 35.9 for standard RAG. Our frozen reader scores only 46.9 EM on 2Wiki even with the gold passages, so the gap lies in the reader and the setup rather than in retrieval. The 65.0 EM quoted for HotpotQA is our train-split number at k = 20.
Q2 ("$0 per query"): We agree and will reword it to "zero generative calls". The measured cost is \0.05per1,000queries,unbatchedonanH200,whichisabout0.015 above ColBERTv2+CE, plus \0.21to0.36 per index. On the same GPU IRCoT costs \0.95to1.18 per 1,000 queries, or \0.26to0.37 through an API. Further details are in G2.
Q4 (Stronger first-stage seeds): Yes, and thank you for suggesting it. Seeding from the cross-encoder's top 5 improves every set and fixes the depth failure (G3).
Q3 (GRAFT-Net and PullNet): Both are trained and work over a knowledge base. Please see our reply to Reviewer R2 for the full comparison.