- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CHASE, a training-free framework for multi-hop dense retrieval that generates hypothetical supporting facts (HSFs) from questions using an LLM. These HSFs serve as independent retrieval queries, whose results are combined through max-score fusion and anchor-aware reranking without requiring explicit graph construction. Experiments on three multi-hop QA benchmarks demonstrate substantial improvements over standard dense retrieval, while additional analyses investigate representation gaps, generation quality, and model capability requirements.
Critical strengths
(1) The paper addresses an important limitation of multi-hop dense retrieval by explicitly modeling hop-specific information needs without requiring graph construction or retriever training. (2) The experimental design includes controlled comparisons under identical retrieval settings, along with extensive ablations that effectively isolate the contributions of hypothetical supporting facts, fusion strategies, and reranking. (3) The analysis of embedding-space representation gaps and generator capability provides useful insights into why the approach works and under which conditions its effectiveness deteriorates.
Critical weaknesses
(1) The effectiveness of CHASE heavily depends on a powerful proprietary LLM for generating high-quality supporting facts. Smaller open-source models frequently fail to outperform question-only retrieval, limiting the approach's practical accessibility. (2) Despite using only one LLM call per question, the method incurs substantial generation overhead, requiring approximately 9K–16K tokens per query. This raises concerns about deployment efficiency and scalability in high-volume retrieval scenarios. (3) The proposed approach builds primarily on existing techniques such as hypothetical document generation, multi-query retrieval, and score fusion. Its methodological novelty is therefore somewhat limited, particularly since simpler interrogative sub-queries outperform HSFs on 2WikiMultiHopQA.
Questions for authors
Can the authors investigate whether shorter prompts or smaller distilled generators can preserve CHASE's retrieval improvements while significantly reducing its inference cost?
2 comments
Response to Reviewer oZeW (2/2)
W3 — Methodological novelty, particularly since sub-queries outperform HSFs on 2Wiki
You are right, and the new controls make the point sharper than the submitted version did. We ran two experiments that bear on it.
First, a matched HyDE baseline (same generator, encoder, index, budget), which the submitted paper lacked:
| Recall@2 | MuSiQue | 2Wiki | HotpotQA |
|---|---|---|---|
| Question only | 44.12 | 63.88 | 71.30 |
| HyDE (one global hyp. doc.) | 49.12 | 62.63 | 72.05 |
| CHASE | 58.04 | 74.97 | 78.30 |
HyDE loses 1.25 points on 2Wiki and trails CHASE by 8.92 / 12.34 / 6.25. The separation from the nearest generation-based precedent is therefore large and not an artifact of multi-query retrieval in general.
Second, on your specific observation about 2Wiki: we removed the confound in our sub-query comparison (the old prompt let HSFs guess entities but restricted sub-queries to entities named in the question). With information content held fixed, the gap is +2.09 / −1.35 / +3.00, and under anchor-aware reranking it disappears entirely, with the two forms within 0.60 points on every benchmark.
Rather than defend declarative form, we have restated the contribution. Reranking the sub-query pool against its own anchors gains 10.00 / 12.10 / 3.70 over question-only reranking of the same 80-candidate pool. Anchor-aware reranking is thus a contribution independent of anchor form, the one component whose benefit is uniform across benchmarks, and it now appears as a named contribution in §1 rather than an optional stage. The claim the paper makes is that anchors must be hop-local and instantiate omitted entities, and that candidates should be scored against their best-matching anchor; phrasing is a per-benchmark tuning choice.
The practical answer to your question is that zero-shot prompting with a flash-tier model costs about 2K tokens per question and stays at or above our frontier-model configuration on two of three benchmarks. If this addresses your concerns, we would be grateful if you would consider revisiting your assessment.
Response to Reviewer oZeW (1/2)
Thank you. You asked whether shorter prompts or smaller generators could preserve CHASE's gains while cutting inference cost. We tested both, and both work, which addresses the deployment objection in your W1 and W2.
Q — Can shorter prompts or smaller/distilled generators preserve the gains?
Shorter prompts: yes, almost free. We varied demonstrations per question type from five to zero, holding generator, encoder, corpus and index fixed. The zero-shot condition keeps only the system prompt, which contains no dataset-specific content and no examples.
Going from five demonstrations to zero costs 1.77 / 1.58 / 1.30 Recall@2 while cutting tokens 8.9× / 5.8× / 8.7×, to roughly 2,000–2,300 per question, and still clears question-only retrieval by 13.9 / 11.8 / 10.5 points. The demonstrations buy the last point or two of accuracy; they are not what makes the method work.
Cheaper generators: yes, correcting our W1 claim. A flash-tier hosted model (DeepSeek-v4.1-flash) matches or exceeds our frontier-model configuration on all three benchmarks:
The advantage persists after anchor-aware reranking (+0.78 / +0.88 / +1.20).
Combined, this answers your W1 and W2 together. The submitted paper claimed the method "rests on a frontier-scale generator." That claim was wrong, and we have withdrawn it. A flash-tier model prompted zero-shot reaches 58.04 / 75.70 / 81.80 Recall@2, at or above our frontier-model configuration on two of three benchmarks, at roughly one ninth of the token cost. It retains 88–89% of the same generator's five-shot gain, and exceeds the gain of our frontier-model main configuration on MuSiQue and 2Wiki. The open models below 35B fail for want of instruction-following reliability, not parameter count: the task needs a well-formed, entity-instantiated chain, and a cheap hosted model that emits one reliably suffices. We have rewritten §4.7, §4.8 and the conclusion on this basis. Appendix F.
The residual ~2K tokens is also the part most amenable to prefix caching, since the prompt prefix is identical for every question, a saving we do not exploit in the reported numbers.