- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses two main problems in LLM search-based reasoning: repeatedly solving the same subproblems and making errors that lead to wasted search effort and incorrect answers. The authors organize reasoning into a shared AND-OR graph and introduce Active Shortcut Discovery. The method combines LLM-generated candidate actions with subproblem reuse, structured search, and verification, allowing previous reasoning results to be reused in future problem solving. Experiments show substantial improvements in Macro-F1 on two reasoning benchmarks.
Critical strengths
- The problem is clearly defined. The paper distinguishes redundancy within a single problem instance from redundancy across instances, and reuses concrete intermediate results through shared states. This is more direct than storing strategies or prompt templates: the system records how far a subproblem has been explored and whether it has been solved.
- Including unsolvable instances strengthens the evaluation. Testing only solvable problems can hide a model’s tendency to produce an answer even when none exists. WickedGame24 and the boolean expression task include unsolvable or malformed inputs, testing both the ability to find correct solutions and to reject invalid instances. In Table 2, the proposed method achieves low rates across all three error categories, which is more informative than overall accuracy alone.
Critical weaknesses
- The ablation studies are not comprehensive enough. Table 3 examines verification and shortcut discovery, but the provided sections lack separate ablations of state merging and cross-instance reuse. It remains unclear whether the gains mainly come from verification, a better search strategy, or shared states. Without verification, adding shortcuts reduces token usage but lowers Macro-F1 from 28.3 to 21.9, suggesting that shortcuts depend on reliable intermediate results and may amplify errors.
- The comparison involving cross-instance sharing may not be entirely fair. The proposed method shares search results across multiple instances, whereas some baselines mainly solve each instance independently.
Questions for authors
- How does the method ensure that all valid actions have been explored, rather than the model failing to propose the correct action or the search budget running out? The paper states that fully explored states can be pruned, but the experiments use limited visit budgets and model sampling to generate actions.
- Do the baselines have access to the same historical information and opportunities for reuse? Are tools, verifiers, and computational budgets consistent across methods?