acceptodds
Under review as a conference paper at ICLR 2027

Beyond Logical Decomposition: Prioritizing Retrieval Actions in Multi-Hop QA

Abstract

Multi-hop retrieval-augmented generation systems typically plan retrieval by the logical dependencies in a question, but logically valid retrieval actions can differ greatly in how much they narrow the search. We study this mismatch as the Search-Priority Gap: when several valid actions are available, current planners do not reliably start from the most selective one. To isolate this behavior, we construct 2,541 question pairs from 2WikiMultiHopQA by adding true but less selective facts about the bridge entity from Wikidata, while keeping the original identifying relation, the anchor, and thus the answer unchanged. Across decomposition-based, iterative, and agentic retrieval systems, these added facts cause substantial accuracy drops, e.g., from 67.1% to 31.0% for PlanRAG. The drop depends on which facts are added and on where the anchor appears in the question, even though its position does not change the problem. Trajectory analysis shows that added facts pull planners away from the anchor, so the bridge is found late or not at all. A simple few-shot prompt that prioritizes the anchor recovers part of the drop, raising AceSearcher-14B accuracy from 50.8% to 66.7%. These results suggest that multi-hop retrieval planning should account not only for logical dependency, but also for the retrieval priority of candidate actions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Hop Question Answering, Retrieval-Augmented Generation, Retrieval Planning, Query Decomposition, Multi-Hop Retrieval, Retrieval Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fRf3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper highlights a major blind spot in multi-hop AI search systems: they get easily distracted. Even when they're following a logical path, they don't always tackle the most specific, helpful steps first, which is a flaw the authors call the "search priority gap." When researchers tested this by throwing in extra, factually true but less relevant details, the AI systems got bogged down, missed the crucial clues (the "anchors"), and saw their accuracy completely tank. The good news is that there's a straightforward fix. Just giving the AI a simple prompt that trains it to focus on those key anchors first brings its performance right back up. Ultimately, it shows that we need to teach these AI planners exactly what to prioritize in a search, rather than just letting them follow basic logic.

Critical strengths

This paper has some strong points. First, it introduces the idea of a "search-priority gap," pointing out that just because a search is logically valid doesn't mean it's efficient, which gives a fresh perspective on search planning. The setup is also rigorous, instead of just relying on existing datasets, they built 2,541 controlled question-answer pairs to pinpoint exactly how retrieval priority impacts system performance. And, they tested a great number of baselines, covering everything from standard parametric models to prompt-based and specialized retrieval systems. Best of all, they didn't just stop at noting performance degradation; they really dug into the trajectory analysis to figure out the actual failure mechanisms behind the scenes.

Critical weaknesses

Here are a few simple ideas to make the paper even better:

  1. It would be great to test on more than just two-step questions. Trying it out on longer questions or in specific areas like medicine or finance would show that this search problem happens everywhere.
  2. The way you measure search difficulty could be more realistic. Instead of just looking at the knowledge graph, matching it with how often words actually show up in the text would make a lot more sense for real-world searches.
  3. Using prompts to fix the issue is a good start, but building a real system update would be a huge plus. Training the model to automatically plan its searches better from the ground up would be a much bigger contribution.
  4. You need to make sure the prompt fix worked because of the search order and not just because of the extra examples. Running a test where everything is exactly the same except the search order would prove your point perfectly.
Questions for authors

All my question in the weaknesses.

3 comments

We sincerely thank the reviewer for the thoughtful and constructive suggestions. We appreciate the opportunity to strengthen our paper in response to these comments. Specifically, we have added a text-based selectivity analysis and a controlled prompt experiment, and further clarified our experimental scope, limitations, and future directions. We address each point below.

W1. Longer reasoning paths and domain-specific evaluation

We agree that longer reasoning paths and specialized domains would help establish how broadly this problem occurs. We focus on two-hop questions because 2WikiMultiHopQA provides the gold intermediate entities and Wikidata IDs needed to construct valid constraints while preserving the original reasoning path and answer. This allows us to isolate retrieval prioritization without introducing errors from entity linking or longer reasoning chains. We have clarified this choice in Appendix B.1, and we discuss extensions to longer questions and domains such as medicine and finance in Sec. 5.

W2. Measuring selectivity in text retrieval

We thank the reviewer for this valuable suggestion. We agree that selectivity should also be measured in the text that is actually searched. For every added fact in our rewritten questions, we counted how many passages in our retrieval corpus mention it, and compared this corpus frequency with the fact's Wikidata selectivity category from App. B.2.

CategoryκnCorpus freq.
VS≤ 1082293
S11–100907361
M101–1,000497598
NS> 1,0008,94311,171

κ is the number of entities that satisfy the fact in Wikidata, n is the number of added facts, and Corpus freq. is the mean number of passages that mention the fact.

The two measures align: the less selective a fact is in Wikidata, the more passages mention it, from 293 on average for the most selective facts to 11,171 for non-selective ones. Selectivity measured in the text therefore gives the same picture as our Wikidata-based measure.

W3. Improving the retrieval planning system

We appreciate this suggestion and agree that developing a general retrieval planning method would be a valuable next step. Our current work focuses on identifying and understanding the problem rather than proposing a new planning system. The few-shot experiment shows that changing retrieval planning alone can improve accuracy without retraining or changing the retriever. We hope these findings can guide such methods, and we discuss ranking candidate retrieval actions, replanning, and training signals for search policies in Sec. 5.

W4. Isolating the effect of retrieval order from few-shot examples

We thank the reviewer for this important suggestion. Our ablation shows that the improvement is not simply due to adding few-shot examples. How these examples guide the model to choose its first retrieval action also matters.

In our setting, few-shot examples are needed to keep the decomposition format consistent for controlled retrieval. Since our original examples also guide the model to search the anchor first, removing them would change both the format and the retrieval strategy. To separate these factors, we keep the same four example questions and output format but change only their first sub-question. In Ours, Q1 searches the anchor alone (e.g., "Who directed film X?"). In Reordered, Q1 searches the added constraints (e.g., "Who received award Y, attended school Z and direct film X?"). The second sub-question remains unchanged.

The table reports accuracy (%) on rewritten questions. Base and Ours are from Table 3, while Reordered uses the modified examples.

BaselineBaseOursReorderedΔ
Vanilla RAG17.064.428.6−35.8
AceSearcher-1.5B43.157.350.5−6.8
AceSearcher-8B55.458.247.6−10.6
AceSearcher-14B50.866.749.6−17.1
AceSearcher-32B50.367.153.6−13.5

Accuracy decreases for all five models when the examples guide the model to search the added constraints first. For example, Vanilla RAG drops from 64.4% to 28.6%, and AceSearcher-14B drops from 66.7% to 49.6%. Since the same four example questions are used in both settings, these results support that the improvement depends on guiding the first retrieval action toward the anchor, rather than simply providing additional examples.

We hope these responses address the reviewer's concerns, and we are happy to discuss any remaining questions.

Reviewer_fRf3OpenReview ↗

Response to the rebuttal

Thank you to the authors for the response and I'm happy to maintain my positive assessment.

Thank you for your positive feedback and for taking the time to consider our response. We appreciate your support and thoughtful evaluation of our work.

Reviewer_uNGwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how multi-hop RAG systems choose what to search for when a question contains several valid clues about the same bridge entity. This study creates 2,541 pairs of questions by adding true but less selective facts about the bridge entity while maintaining the initial anchor and answer constant. Large reductions in accuracy have been found for different RAG models.

Critical strengths
  1. This paper examines an underexplored issue with multi-hop RAG. Even though the additional facts are correct, and the original reasoning path is preserved, the accuracy of such models has degraded quite a lot.
  2. The experiments done without retrieval aid us in identifying the source of such degradation. Qwen2.5-14B has performed slightly better on the rewritten questions without the use of retrieval, while Vanilla RAG with the same backbone has performed very poorly. This suggests that the issue is not simply that the rewritten questions are more challenging to answer.
  3. The paper is well-written and clear. Apart from analyzing the final accuracy, the authors have conducted a thorough analysis of the generated queries and intermediate results as well. In particular, PlanRAG and AceSearcher behave differently in their failures, which cannot be determined by just accuracy.
Critical weaknesses
  1. To my mind, the explanations for the accuracy drops are less convincing than the results themselves. As for PlanRAG, 93.4% of first queries contain the anchor (Table 16). Nonetheless, these queries usually contain other facts. The bridge can be found in the first retrieved passages for 97.3% of rewritten questions, while Bri@1 reaches 70.2% (Figure 5). The explanation of the phenomenon has been offered in the paper; however, it is not obvious what part of the problem comes from the queries themselves and what part comes from failing to identify the correct entity among the retrieved passages.
  2. Moreover, the authors consider the anchor to be the best starting point, because it can uniquely identify the bridge in Wikidata. Nevertheless, retrieval is performed over Wikipedia text using E5. There are doubts concerning whether the anchor-only query can provide more relevant evidence than the query with additional facts.
  3. I should highlight my reservations concerning the quality of the rewritten questions. Appendix B.4 mentions that automatic checks cannot guarantee that the original meaning is preserved, and I could not find any reported manual validation. Such an observation can also be made regarding the anchor-position experiment, because the three versions are produced separately.
Questions for authors
  1. Is it possible to test both anchor-only queries and mixed queries against the same questions using the same retriever and give Recall@10 results for the gold first-hop supporting passages?
  2. Have any of the rephrased questions been manually verified? If yes, what was the error rate (including ambiguous descriptions)?
  3. All four examples in Appendix G involve the relations between films and directors. Will these enhancements work on other types of relations? Or with four-shot examples that don’t necessarily start with the anchor?
3 comments

We thank the reviewer for the careful reading and helpful suggestions. We followed the suggestion in Q1 to set up a controlled experiment, and we use it to answer W1 and W2 as well. The experiment shows that the anchor alone retrieves the evidence better than the Mixed query, and that most of the drop comes from identifying the bridge rather than from retrieval.

Experiment setting. Table 16 shows that most first queries of PlanRAG are Mixed, which means they contain the anchor together with added facts. For each Mixed query, we use a template to generate a clean version that keeps only the anchor, for example "Who directed the film Polish-Russian War?". We run both queries on the same question with the same E5 retriever and the same reader. Recall@k is the percentage of questions whose gold first-hop passage is among the top k retrieved passages. Bridge hit is the percentage of questions whose first intermediate answer names the bridge entity, which corresponds to Bri@1 in the paper.

First queryRecall@1Recall@5Recall@10Bridge hit
Anchor-only89.599.199.893.0
Mixed80.195.997.868.9

W1, W2, Q1. Comparing anchor-only and mixed first queries

Retrieval is similar for the two queries, but reasoning with the Mixed query is more challenging. Both queries retrieve the gold first-hop passage in the top 10 for almost all questions, and the anchor-only query is slightly better at every k, so the added facts do not bring more relevant evidence. The difference appears afterwards, when the reader has to answer the sub-query from the retrieved passages. With the anchor-only query, the reader looks for one fact that the gold passage states, and the bridge hit is 93.0%. With the Mixed query, the reader has to find an entity that matches several facts at once, while the retrieved passage usually supports only the anchor fact, and the bridge hit is 68.9%. In 66.4% of PlanRAG's failed first steps, the model states that no entity matches the description, compared with 16.9% on the original questions. The problem therefore comes mainly from identifying the bridge among the retrieved passages, and only to a small extent from the retrieval itself.

The high Recall@10 for both queries can be explained by the retrieval corpus. We use the Wikipedia corpus provided by 2WikiMultiHopQA, which was built for multi-hop question answering rather than large-scale retrieval evaluation. It contains 414,404 passages, and the gold first-hop passage is often easy to find using the anchor alone. Since the Mixed query still contains the anchor, the retriever can usually find the gold passage despite the additional facts.

This relatively clean retrieval setting also helps us isolate the effect of the first query on bridge identification. Even when the correct passage is retrieved, changing the first query from Anchor-only to Mixed substantially reduces the model's ability to identify the bridge. In a larger open-domain corpus, additional facts could introduce more competing passages and cause greater retrieval degradation. We have not tested this possibility and acknowledge it as a limitation in Section 5.

W3, Q2. Manual validation of rewritten questions

We thank the reviewer for this suggestion, and we have added a manual validation to further show the quality of our rewritten questions. The validation confirms that 99.4% of the rewrites are valid, with an observed error rate of 0.6%. Three annotators evaluated 334 sampled rewrites, which corresponds to a 95% confidence level with a 5% margin of error. They checked whether each question keeps the bridge hidden, preserves the original anchor, and retains the correct answer. By majority vote, 332 of the 334 rewrites pass all three checks, and annotator agreement is at least 95%. Fluency is 85.0%, mainly due to repetitive descriptions, but most of these cases still preserve the original meaning. We also clarify that the anchor-position variants were generated separately and may have minor wording differences. These details are added to Sec. 3.1 and Appendix B.

Q3. Film–director relations in the four-shot examples

We thank the reviewer for pointing out the topics of the few-shot examples. The focus on films follows the topic distribution of the original dataset. We manually selected representative examples from the most common question type, and the director relation is the anchor of 76.0% of our evaluation questions. To make this distribution clear, we added a topic analysis in Table 7 of Appendix B.6. The original 2WikiMultiHopQA dataset is dominated by People & Biography, Art & Culture, and History. After selecting eligible questions, Art & Culture at 39.1% and Music at 24.5% become the most common topics. Our four-shot examples were therefore selected from this common question pattern.

We hope these responses address the reviewer's concerns, and we are happy to discuss any remaining questions.

Thanks for the detailed response. My concerns have been addressed, and I will maintain my positive rating.

Thank you for your encouraging comments. We're glad our response addressed your concerns, and we appreciate your careful review of our work.

Reviewer_dj6kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the Search-Priority Gap in RAG, where retrieval planners fail to prioritize effective retrieval actions among multiple alternatives. The paper introduces a specific dataset by adding some less selective constraints about bridge entities. Their controlled experiments, which span across decomposition-based, iterative, and trained retrieval agents demonstrate accuracy degradation on their rewritten questions. The authors further analyze the effects of constraint selectivity, anchor position, and retrieval trajectories to investigate where failures occur.

Critical strengths
  • Important and interesting perspective: the priority gap is real and important under the dense retrieval setting that harms efficiency and accuracy at the same time.
  • Systematic analysis: The paper extensively evaluates across different retrieval paradigms, as well as the analysis of anchor usage, passage coverage, and bridge identification, provides useful insights into how failures occur.
Critical weaknesses
  • Unclear significance under stronger retrieval planning: Although the paper demonstrates substantial performance degradation across several RAG systems, it remains unclear whether this reflects a fundamental challenge in retrieval-action prioritization or limitations of the evaluated planning strategies. For example, simple anchor-first prompting nearly eliminates the performance gap for Vanilla RAG without modifying the dense retrieval backend. These observations might suggest that the failure may be largely mitigated through better retrieval planning alone.
Questions for authors
  • Do authors try to find whether more modern QA datasets (for RAG) already cover similar situations? For example, hard negatives or distractors in the query.
  • Do the propose gap still exist if we move beyond dense retrieval? For example, for more modern agentic retrieval with more capable model backbone, query re-writing/refinement might gives the correct ordering given the agent's own knowledge about the selectivity among candidates is better.
5 comments

We sincerely thank the reviewer for the thoughtful comments and helpful questions. We would like to first clarify the main focus of this paper and the experimental setup, and then respond to the weaknesses and the questions below. Our study focuses on the planning stage, where the model decides which retrieval action to take, before the retriever executes the chosen query. All systems use the same E5-large-v2 dense retriever and shared FAISS index to isolate differences in planning. When several valid retrieval actions are available, a good plan should start from the most selective one. Our key investigation is whether current planning strategies can make this choice.

W1. Significance under stronger retrieval planning

We appreciate this concern and would like to clarify the scope of our study. Retrieval-action prioritization is the capability of planning strategies that we evaluate. We evaluate a broad set of state-of-the-art RAG systems, including established planning-based methods (IRCoT, DecomP-ODQA, PlanRAG, and RT-RAG), state-of-the-art RL-trained search agents (Search-R1 and AceSearcher up to 32B), and the advanced closed-source GPT-5.6 Terra agent. All of them show accuracy drops on rewritten questions in Table 1. For example, Search-R1-7B drops from 70.9% to 57.9%, and AceSearcher-14B from 74.0% to 50.8%. GPT-5.6 Terra is more robust, dropping only from 79.7% to 75.4%, but it starts from the most selective anchor alone in only 49.2% of rewritten questions. Its reasoning tokens also increase 8.9×, from 72K to 643K, as reported in Table 11. These results suggest that stronger planning helps, while reliably choosing the best retrieval action remains a challenge.

W2. Improvement from anchor-first prompting

We appreciate the reviewer highlighting the effectiveness of our simple prompt-based mitigation. This result is consistent with the main finding of our study. Our work investigates how models prioritize retrieval actions, and the improvement from anchor-first prompting supports the importance of this choice. By guiding the model to start from the most selective anchor, we improve accuracy without changing the retriever or retraining the model.

We would like to kindly clarify that the gap remains for strong trained models even with anchor-first prompting, as shown in Tables 1 and 3. With the prompt, AceSearcher-14B reaches 66.7% on rewritten questions, up from 50.8% but still below its 74.0% on the original questions. AceSearcher-32B reaches 67.1%, up from 50.3% and still below its original 71.9%. All four AceSearcher variants stay below their original accuracy. Prompting therefore reduces the gap but does not remove it.

This result also points to a direction for future work. Current methods mainly improve planning through training, such as reinforcement learning. Our findings suggest that once the gap is clearly identified, a prompt alone can already be surprisingly helpful. This is consistent with recent work such as GEPA [1], which shows that careful prompt optimization can outperform RL-based training such as GRPO.

Q1. Coverage in existing QA benchmarks

To the best of our knowledge, existing QA benchmarks do not explicitly evaluate this setting. HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle typically require following a reasoning chain to identify the bridge entity, rather than choosing among multiple valid constraints about that entity. More challenging benchmarks, such as ComplexWebQuestions, FRAMES, and BrowseComp, contain multiple constraints, but they are not designed to test which constraint should be searched first. In our setting, the original identifying relation is already sufficient to find the bridge entity. We deliberately add other true but less selective constraints about the same entity, creating alternative valid retrieval actions without changing the answer. This also differs from hard negatives or distractors, which add misleading or irrelevant content, while our added constraints are all correct. This allows us to isolate whether models prioritize the most effective action. We clarify this distinction in Sec. 2 and Appendix A.1.

Q2. Stronger agentic retrieval and retrieval backends

We thank the reviewer for this question. As discussed in our response to W1 above, the gap also appears for stronger agentic models that iteratively search and refine queries, including Search-R1, AceSearcher, and GPT-5.6 Terra. Regarding retrieval beyond dense retrieval, we keep a single dense retriever in this study to isolate planning. We agree that other backends are a valuable extension, and we note this in the Limitations in Sec. 5.

We hope these responses address the reviewer's concerns, and we are happy to discuss any remaining questions.

References

[1] Agrawal, Lakshya A., et al. "Gepa: Reflective prompt evolution can outperform reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.

Dear Reviewer,

We would like to kindly follow up on our rebuttal and gently remind you to review our responses when you have a chance. We sincerely appreciate your time, thoughtful feedback, and consideration.

Thank you again for your valuable effort and time.

Thank you for the rebuttal and additional analyses.

After reviewing your response, I remain unconvinced by the paper’s central claim: the observed degradation reflects "Search-Priority Gap". Specifically, I see some conceptual issues:

  • Misattribution of the Failure Mechanism: The paper claims that failing to prioritize selective actions causes retrieval to drown in a massive candidate space. However, your own data shows that PlanRAG and Search-R1-7B-Base retain the anchor in ~90–93% of first queries and retrieve the gold passage at step 1 with 97.3% (Psg@1) / 97.8% (Recall@10) and 92.3% (Psg@1), respectively. The search step itself succeeds. The sharp drop occurs after retrieval because, as you explicitly note in the discussion, "the reader has to find an entity that matches several facts at once, while the retrieved passage usually supports only the anchor fact," causing the reader to abstain.
  • Conflating Redundant Query Stuffing with Multi-Constraint Planning: Unlike genuine multi-constraint QA where multiple clues are needed to intersect candidate sets, the proposed benchmark stuffs redundant info into questions where a single anchor is already uniquely identifying. Penalizing planners for faithfully incorporating explicit user constraints conflates sensitivity to redundant modifiers with a planning failure.

Given these main flaws, I might consider decreasing my score. I would also call for other reviewers' and AC's attention if my concern holds. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.