- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CAPS, an efficient pairwise verification framework for parallel reasoning. It uses a cascaded selection strategy that progressively allocates more verification compute to promising candidates through partial-evidence filtering, full-evidence comparison, and final round-robin selection. Experiments on math and code benchmarks demonstrate that CAPS can generally maintain or improve selection performance while significantly reducing verification cost compared with existing pairwise methods.
Critical strengths
- The general design of the proposed method is intuitively reasonable. The idea of using cheaper partial evidence for early filtering and reserving more expensive full-evidence comparisons for stronger candidates is well motivated and aligns well with the goal of reducing verification cost.
- The ablation study is relatively comprehensive. The paper examines several important components of the proposed framework, including partial-evidence filtering, pairing strategy, rescue mechanism, and candidate pool size, which helps clarify where the performance and efficiency gains come from.
Critical weaknesses
- The evaluation size for the mathematical tasks is relatively small. AIME 2025 and HMMT 2025 contain only 30 examples each. It is suggested to include a larger-scale mathematical reasoning benchmark, such as Omni-MATH or MATH-500, to provide more statistically reliable evidence for the effectiveness of the proposed method.
- The performance improvement over the strongest baseline is relatively modest. While CAPS substantially reduces verification cost, its accuracy gains over V1-Infer are generally small and are not consistent across all settings.
- Given that the proposed method uses partial evidence to estimate candidate quality before performing full verification, it is suggested to compare with tree-search-based reasoning methods, such as [1], where partial trajectories are directly evaluated to guide pruning and compute allocation during generation.
- The design of partial evidence is somewhat heuristic. The method uses a fixed reasoning window for math problems and the first few lines of code for programming tasks. These choices are manually specified and may be coarse, since different examples may require different amounts or types of evidence to reliably assess solution quality.
[1] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., & Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36, 11809-11822.
Questions for authors
- Can the authors provide more details on how the rescue threshold is selected?
Thank you for your time and feedback - happy to discuss further
Dear Reviewer nw9U,
Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.
Thank you again for your time and consideration.
Best regards, Authors of Submission #52086