- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper highlights a major blind spot in multi-hop AI search systems: they get easily distracted. Even when they're following a logical path, they don't always tackle the most specific, helpful steps first, which is a flaw the authors call the "search priority gap." When researchers tested this by throwing in extra, factually true but less relevant details, the AI systems got bogged down, missed the crucial clues (the "anchors"), and saw their accuracy completely tank. The good news is that there's a straightforward fix. Just giving the AI a simple prompt that trains it to focus on those key anchors first brings its performance right back up. Ultimately, it shows that we need to teach these AI planners exactly what to prioritize in a search, rather than just letting them follow basic logic.
Critical strengths
This paper has some strong points. First, it introduces the idea of a "search-priority gap," pointing out that just because a search is logically valid doesn't mean it's efficient, which gives a fresh perspective on search planning. The setup is also rigorous, instead of just relying on existing datasets, they built 2,541 controlled question-answer pairs to pinpoint exactly how retrieval priority impacts system performance. And, they tested a great number of baselines, covering everything from standard parametric models to prompt-based and specialized retrieval systems. Best of all, they didn't just stop at noting performance degradation; they really dug into the trajectory analysis to figure out the actual failure mechanisms behind the scenes.
Critical weaknesses
Here are a few simple ideas to make the paper even better:
- It would be great to test on more than just two-step questions. Trying it out on longer questions or in specific areas like medicine or finance would show that this search problem happens everywhere.
- The way you measure search difficulty could be more realistic. Instead of just looking at the knowledge graph, matching it with how often words actually show up in the text would make a lot more sense for real-world searches.
- Using prompts to fix the issue is a good start, but building a real system update would be a huge plus. Training the model to automatically plan its searches better from the ground up would be a much bigger contribution.
- You need to make sure the prompt fix worked because of the search order and not just because of the extra examples. Running a test where everything is exactly the same except the search order would prove your point perfectly.
Questions for authors
All my question in the weaknesses.
3 comments
Response to the rebuttal
Thank you to the authors for the response and I'm happy to maintain my positive assessment.
Thank you for your positive feedback and for taking the time to consider our response. We appreciate your support and thoughtful evaluation of our work.
We sincerely thank the reviewer for the thoughtful and constructive suggestions. We appreciate the opportunity to strengthen our paper in response to these comments. Specifically, we have added a text-based selectivity analysis and a controlled prompt experiment, and further clarified our experimental scope, limitations, and future directions. We address each point below.
W1. Longer reasoning paths and domain-specific evaluation
We agree that longer reasoning paths and specialized domains would help establish how broadly this problem occurs. We focus on two-hop questions because 2WikiMultiHopQA provides the gold intermediate entities and Wikidata IDs needed to construct valid constraints while preserving the original reasoning path and answer. This allows us to isolate retrieval prioritization without introducing errors from entity linking or longer reasoning chains. We have clarified this choice in Appendix B.1, and we discuss extensions to longer questions and domains such as medicine and finance in Sec. 5.
W2. Measuring selectivity in text retrieval
We thank the reviewer for this valuable suggestion. We agree that selectivity should also be measured in the text that is actually searched. For every added fact in our rewritten questions, we counted how many passages in our retrieval corpus mention it, and compared this corpus frequency with the fact's Wikidata selectivity category from App. B.2.
κ is the number of entities that satisfy the fact in Wikidata, n is the number of added facts, and Corpus freq. is the mean number of passages that mention the fact.
The two measures align: the less selective a fact is in Wikidata, the more passages mention it, from 293 on average for the most selective facts to 11,171 for non-selective ones. Selectivity measured in the text therefore gives the same picture as our Wikidata-based measure.
W3. Improving the retrieval planning system
We appreciate this suggestion and agree that developing a general retrieval planning method would be a valuable next step. Our current work focuses on identifying and understanding the problem rather than proposing a new planning system. The few-shot experiment shows that changing retrieval planning alone can improve accuracy without retraining or changing the retriever. We hope these findings can guide such methods, and we discuss ranking candidate retrieval actions, replanning, and training signals for search policies in Sec. 5.
W4. Isolating the effect of retrieval order from few-shot examples
We thank the reviewer for this important suggestion. Our ablation shows that the improvement is not simply due to adding few-shot examples. How these examples guide the model to choose its first retrieval action also matters.
In our setting, few-shot examples are needed to keep the decomposition format consistent for controlled retrieval. Since our original examples also guide the model to search the anchor first, removing them would change both the format and the retrieval strategy. To separate these factors, we keep the same four example questions and output format but change only their first sub-question. In Ours, Q1 searches the anchor alone (e.g., "Who directed film X?"). In Reordered, Q1 searches the added constraints (e.g., "Who received award Y, attended school Z and direct film X?"). The second sub-question remains unchanged.
The table reports accuracy (%) on rewritten questions. Base and Ours are from Table 3, while Reordered uses the modified examples.
Accuracy decreases for all five models when the examples guide the model to search the added constraints first. For example, Vanilla RAG drops from 64.4% to 28.6%, and AceSearcher-14B drops from 66.7% to 49.6%. Since the same four example questions are used in both settings, these results support that the improvement depends on guiding the first retrieval action toward the anchor, rather than simply providing additional examples.
We hope these responses address the reviewer's concerns, and we are happy to discuss any remaining questions.