- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RoHoper, a general prompting strategy that can help in reasoning. Instead of following a fixed plan, the model generates several intermediate questions, answers one of them, and then updates its plan based on the new information. Then it uses multiple reasoning paths and majority voting to produce the final answer. The proposed method shows improvement on 3 datasets across many LLMs.
Critical strengths
-
The proposed method is simple and easy to use. It does not require task-specific training or examples. The authors also test it on multiple LLMs.
-
Detailed analysis. The paper provides many useful analyses, including how the number of reasoning paths and steps affects performance.
-
Covered different prompting strategies. The paper covers several common prompting strategies.
Critical weaknesses
-
Limited novelty. I have some concerns about the novelty of the proposed method. Replanning is already a common idea in LLM agents: the agent first makes a plan, executes one step, and then updates its plan based on new information. ReHoPER mainly applies this idea to generating and answering intermediate questions. However, I think generating new intermediate questions is also a form of planning, so the main idea does not seem very inspiring. I suggest comparing against agentic planning methods, such as RAFA [1], or other methods that dynamically update their plans. I understand this paper also takes the way about how to aggregate these paths as a new idea, but I believe it is still worth exploring if deleting the question regeneration and making it an easy replanning will make any difference.
-
Limited diversity of reasoning benchmarks. The paper evaluates on MuSR, MoreHopQA, and iLLC (a new dataset from the author). While testing many models is a strength, I think the number and diversity of datasets are still too limited. In particular, iLLC mainly tests string operations, which may not be that useful. Actually, for the other prompting papers, they usually include many other tasks, such as mathematical and symbolic reasoning. Moreover, the improvements on MoreHopQA are not always positive across models and prompting strategies (Table 1). Besides, based on the LLM's ability, I think evaluating on more challenging QA benchmarks on some advanced LLMs, such as BrowseComp-Plus [2], would make the paper more impactful, especially if the method can show improvements on these more complex tasks.
[1] RAFA: Reason for Future, Act for Now. International Conference on Machine Learning (ICML 2024). https://proceedings.mlr.press/v235/liu24ab.html
[2] BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent. https://arxiv.org/abs/2508.06600
Questions for authors
-
Does replanning really improve the reasoning process, or does it mainly increase answer diversity? Compared with Self-Consistency, I wonder whether the improvements come from better reasoning plans or simply from generating more diverse answers. I think this is an interesting question worth further discussion and analysis, especially to see if this could be extended in self-improvement.
-
How well does ReHoPER work without Self-Consistency? In practice, we often do not use multiple reasoning paths or majority voting because of the inference cost. What happens if ReHoPER uses only one reasoning path and returns the final answer directly, without majority voting?
4 comments
Part 2 / 2
Q1. We thank the reviewer for this interesting question. We believe the improvements come from both better reasoning and answer aggregation, rather than simply increased answer diversity. In ReHoPER, each prediction is generated using only the original input and the intermediate question-answer history of its own path. The model does not have access to previous predictions from the same path or predictions from other paths. Therefore, later predictions cannot benefit from observing earlier predictions.
As shown in Figure 2 (top row) in the paper, the mean correctness of individual predictions, or Current-Step Mean (CSM), generally increases as more intermediate questions are answered, even before applying any majority voting. This suggests that the evolving reasoning process improves the quality of individual predictions, independently of aggregation.
Furthermore, Table 2 directly examines the contribution of planning. When we remove replanning or look-ahead while keeping the prediction and aggregation procedures unchanged, performance generally decreases. Full ReHoPER outperforms the stronger ablation by 1.9, 2.7, and 2.5 percentage points under IO, CoT, and PS, respectively. These findings support the benefit of adaptive replanning beyond simply generating more diverse answers. We will clarify this distinction and better highlight these analyses in the paper.
Q2. ReHoPER can also operate with a single reasoning path, returning its final prediction directly without majority voting. In fact, our analysis in Figure 2 and Appendix B already provides evidence for this setting through Current-Step Mean (CSM), which measures the accuracy of individual paths without aggregating their predictions. Since the paths are independent, this corresponds to the expected accuracy of a single-path configuration.
Across the 63 model-dataset-prefix configurations analyzed, the final CSM achieves 55.5% accuracy, compared with 64.6% for the full cumulative-voting approach. While aggregation provides an additional performance boost, Figure 2 also shows that individual predictions generally improve as more intermediate questions are answered, even without voting. This suggests that ReHoPER's reasoning process can improve individual predictions independently of Self-Consistency. In addition, using only one path also eliminates the inference cost of the other five paths, providing a lower-cost alternative. We will clarify this distinction and better highlight these results in the revised paper.
Once again, we appreciate your detailed feedback and hope our revisions and clarifications address your concerns. Thank you for contributing to the improvement of our work.
Thank you for the detailed response and the additional experiments.
-
On novelty: At each step, ReHoPER generates the full list of remaining intermediate questions, answers only the first one, and then regenerates the list based on the accumulated question-answer history. This is essentially receding-horizon planning: plan the full trajectory, execute only the first step, observe the result, and replan. RAFA follows the same idea. The main difference is that RAFA relies on external state and reward feedback, while ReHoPER relies on the model’s own evaluation. I agree this is a difference in the problem setting, but I do not think it is a fundamental difference in the method.
-
On BrowseComp-Plus: Since Self-Consistency already performs very well on this subset, it would be more convincing to also try newer and stronger models on harder instances, where there is more room for improvement.
My other concerns have been addressed, and I appreciate the clarifications. Overall, I think this is a good paper with a simple and practical method and thorough analysis. I will keep my positive score.
We thank the reviewer for the follow-up and for their positive assessment of our work.
1. We agree that ReHoPER and RAFA share the receding-horizon planning principle, which we do not claim as novel. Our contribution is applying this principle to task-agnostic, inference-only reasoning through intermediate question answering and prediction aggregation, without external state transitions or reward feedback. Following your suggestion, we also compared ReHoPER against RAFA adapted to our setting across six models:
| Model | MuSR MM RAFA (%) | MuSR MM ReHoPER+IO (%) | MoreHopQA RAFA (%) | MoreHopQA ReHoPER+IO (%) |
|---|---|---|---|---|
| Gemma 3 | 60.4 | 61.2 | 51.3 | 64.7 |
| gpt-oss | 64.8 | 76.0 | 75.3 | 78.7 |
| Llama 3.1 8B | 55.6 | 62.4 | 51.3 | 57.3 |
| Mistral Small | 60.4 | 73.2 | 62.7 | 71.3 |
| Phi-4 | 63.6 | 74.8 | 70.7 | 77.3 |
| Qwen3 | 64.4 | 72.0 | 62.7 | 68.0 |
| Average | 61.5 | 69.9 | 62.3 | 69.6 |
ReHoPER outperforms RAFA in all 12 comparisons, with average improvements of 8.4 pp on MuSR MM and 7.2 pp on MoreHopQA (7.8 pp overall). Together with our ToT comparisons and planning ablations, these results further support the effectiveness of our approach. We will include these results and better clarify the differences from RAFA in the paper.
2. We agree that evaluating stronger models on harder BrowseComp-Plus instances would provide a more convincing comparison. Our preliminary experiments focused on instances with shorter contexts to avoid exceeding the models' context windows. We will extend this evaluation to harder instances with stronger models in the revised work.
We appreciate your constructive feedback and are glad that our previous response addressed your other concerns.
Part 1 / 2
We thank the reviewer for their positive assessment of the clarity, simplicity, and soundness of our approach, and for recognizing its training-free nature, evaluation across multiple LLMs, detailed analysis of reasoning paths and steps, and comparisons with various prompting strategies. We address the concerns below and will incorporate the feedback into the paper.
W1. We agree that replanning and majority voting are established ideas. Our contribution is not either component in isolation. Rather, ReHoPER studies their integration into a task-agnostic, inference-only reasoning procedure. It plans intermediate questions, answers one before replanning from the accumulated question-answer history, and aggregates goal-question predictions across both intermediate steps and independent paths. Our paper already compares ReHoPER against several established approaches, including Chain-of-Thought, Plan-and-Solve, Self-Consistency, Self-Refine, Self-Discover, and Least-to-Most prompting.
While RAFA is related to our work in its planning mechanism, it addresses a different problem setting. RAFA relies on external state transitions and reward feedback to learn environmental dynamics and optimize actions, whereas ReHoPER solves a fixed question using only the input context and model-generated intermediate answers, without external feedback. RAFA also uses task-specific prompts and configurations, so a direct comparison would require adapting its planning and feedback mechanisms to our setting. We will better clarify these differences in the paper.
Based on your feedback, we also conducted additional experiments comparing ReHoPER against Tree of Thoughts (ToT) across six models on MuSR-MM and MoreHopQA. The results are shown below:
ReHoPER outperforms ToT in all 12 comparisons, with average improvements of 5.1 pp on MuSR-MM and 16.6 pp on MoreHopQA, or 10.8 pp overall. These results provide additional evidence of ReHoPER's effectiveness compared to an established search-based reasoning approach.
Finally, Section 5.2 (Table 2) already evaluates the suggested simplifications. -RP generates a fixed list of questions without replanning, while -LA replans by generating only the next question based on the updated history. Both use the same prediction aggregation procedure as ReHoPER. The full approach outperforms the stronger ablation by 1.9/2.7/2.5 pp under IO/CoT/PS, showing that combining look-ahead with replanning improves performance beyond aggregation alone.
W2. We use the same zero-shot planning instructions across three benchmark families that test different forms of multi-step reasoning. MuSR tests narrative deduction, object tracking, and allocation under constraints. MoreHopQA combines evidence across passages and adds arithmetic, symbolic, or commonsense steps (Schnitzler et al., 2024). iLLC extends the last-letter task of Wei et al. (2022) by varying the target character position and the number of words, giving us a controlled test of how models compose a sequence of operations. Focused evaluations of this size have precedents in Reasoning via Planning (Hao et al., 2023), Everything of Thoughts (Ding et al., 2024), and Tree of Thoughts (Yao et al., 2023), which each evaluate three main tasks.
On MoreHopQA, ReHoPER improves average accuracy over Self-Consistency (SC) under all three prompting styles. The gains are 0.8, 2.6, and 2.3 percentage points for input-output, chain-of-thought, and Plan-and-Solve, respectively (Table 1).
To extend the QA evaluation, we followed your suggestion and ran a preliminary experiment on BrowseComp-Plus (Chen et al., 2026). We first took all examples, but some inputs exceeded the models’ context windows even when we supplied only the relevant documents. We then evaluated the 250 instances (a test set comparable in size to those in the submission) with the shortest combined question and document context to avoid this issue.
With the relevant documents supplied, Self-Consistency already performs well on this subset, exceeding 90% accuracy on two of the three models. ReHoPER improves accuracy on all three. For the final paper, we will evaluate a harder subset of BrowseComp-Plus or a more challenging QA dataset to test transfer in a setting with more room for improvement.