acceptodds
Under review as a conference paper at ICLR 2027

CAPS: Cascaded Adaptive Pairwise Selection for Efficient Parallel Reasoning

Abstract

Parallel reasoning, where a generator samples many candidate solutions and an aggregator selects the best, is one of the most effective forms of test-time scaling in large language models, and pairwise self-verification has become its strongest aggregation primitive. Yet pairwise verification carries a heavy cost: each judgment reads two complete solutions in full, and existing methods perform tens of such judgments per problem regardless of whether the comparison is informative. We introduce CAPS (Cascaded Adaptive Pairwise Selection), an inference-only framework that allocates verifier compute non-uniformly along two orthogonal axes: an evidence axis that adapts how much of each candidate the judge sees, and a distribution axis that adapts how comparisons are spread across the pool. CAPS instantiates these into a four-stage cascade with an optional rescue subroutine, and admits a closed-form verifier-token cost in which the per-candidate marginal cost is roughly halved relative to uniform full-evidence schedules. On four self-verifying models (Qwen3-14B, GPT-OSS-20B, Qwen3-4B-Instruct/Thinking) and five reasoning benchmarks spanning code (LiveCodeBench-v5/v6, CodeContests) and math (AIME 2025, HMMT 2025), CAPS outperforms the leading pairwise verifier on 14 of 20 suites while using 25.4% of its verifier-token budget on code, and outperforms pointwise self-verification on all 20. The trade-off suites admit an interpretable diagnostic in terms of the verifier's accuracy at partial versus full evidence, providing a concrete pre-deployment check for cascade suitability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Parallel Reasoning, Large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nw9UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CAPS, an efficient pairwise verification framework for parallel reasoning. It uses a cascaded selection strategy that progressively allocates more verification compute to promising candidates through partial-evidence filtering, full-evidence comparison, and final round-robin selection. Experiments on math and code benchmarks demonstrate that CAPS can generally maintain or improve selection performance while significantly reducing verification cost compared with existing pairwise methods.

Critical strengths
  1. The general design of the proposed method is intuitively reasonable. The idea of using cheaper partial evidence for early filtering and reserving more expensive full-evidence comparisons for stronger candidates is well motivated and aligns well with the goal of reducing verification cost.
  2. The ablation study is relatively comprehensive. The paper examines several important components of the proposed framework, including partial-evidence filtering, pairing strategy, rescue mechanism, and candidate pool size, which helps clarify where the performance and efficiency gains come from.
Critical weaknesses
  1. The evaluation size for the mathematical tasks is relatively small. AIME 2025 and HMMT 2025 contain only 30 examples each. It is suggested to include a larger-scale mathematical reasoning benchmark, such as Omni-MATH or MATH-500, to provide more statistically reliable evidence for the effectiveness of the proposed method.
  2. The performance improvement over the strongest baseline is relatively modest. While CAPS substantially reduces verification cost, its accuracy gains over V1-Infer are generally small and are not consistent across all settings.
  3. Given that the proposed method uses partial evidence to estimate candidate quality before performing full verification, it is suggested to compare with tree-search-based reasoning methods, such as [1], where partial trajectories are directly evaluated to guide pruning and compute allocation during generation.
  4. The design of partial evidence is somewhat heuristic. The method uses a fixed reasoning window for math problems and the first few lines of code for programming tasks. These choices are manually specified and may be coarse, since different examples may require different amounts or types of evidence to reliably assess solution quality.

[1] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., & Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36, 11809-11822.

Questions for authors
  1. Can the authors provide more details on how the rescue threshold δ\delta is selected?
1 comment
AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer nw9U,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #52086

Reviewer_bqoSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CAPS, an inference-only method that reduces the cost of pairwise self-verification for parallel reasoning. CAPS groups candidates by answers or code signatures, runs one elimination round with partial evidence, then reduces the pool with full evidence and runs a round-robin among a few finalists, with an optional rescue step. Experiments are conducted on four self-verifying models across code (LiveCodeBench-v5/v6, CodeContests) and math (AIME 2025, HMMT 2025) benchmarks, comparing against pointwise verification, V1-Infer and majority voting.

Critical strengths
  • Reducing repeated verification of long solutions is a realistic and important problem, and the cascade is a straightforward way to allocate verification effort.
  • The judge-interface and comparison-count controls are valuable. With calls and interface matched, partial evidence reduces input tokens by 35.9% at a cost of 0.5pp on Qwen3-14B/LCB-v6, which is a reasonable accuracy-cost tradeoff.
  • The deployment experiment measures GPU time, throughput and latency, which makes the efficiency claims more concrete than token counts alone.
Critical weaknesses
  • The contribution of each component is not well isolated. The matched-call experiment, which shows that the CAPS procedure itself beats V1-Infer, is only run on one model-benchmark setting. Most of the token savings come from making fewer comparisons (18.5 vs 48 calls). All-E2 is also 0.5-0.7pp better than default CAPS in all ablation settings, so partial evidence mainly reduces cost rather than improving selection. Comparisons against V1-Infer-1× and a plain full-evidence knockout tournament such as Pairwise RM [1] using the same judge interface should be added.
  • The efficiency and accuracy benefits have limited scope. End-to-end savings are much smaller than verifier-token savings: GPU-seconds drop by 13.8% and P50 latency by 3.4% (Table 4), as generation dominates the total compute. For the same reason, the cross-N comparison (CAPS at N = 32 vs V1-Infer at N = 16) is not fair, and results at equal N or equal total compute should be reported instead. On math, CAPS is also not significantly better than majority voting, which does not need judge calls (Table 5).
  • Rescue cannot recover errors from partial-evidence screening in the default setting. With N′ = 16 and f = 4, a candidate that loses outright in Stage A keeps a zero score (Eq. 3), while Stage B losers already have positive scores from their Stage A wins. Since rescue picks the highest-scoring eliminated candidate, it always picks a Stage B loser. The aggregate rescue ablations therefore do not show protection against Stage A errors. This matters most on code, where grouping rarely shrinks the pool.
  • The accuracy aggregation is unclear. LCB-v6 has 131 problems, so a single run changes in steps of about 0.76pp. Values such as 59.4 and 60.1 in Table 10, or 56.0 and 59.1 in Table 2, cannot come from a single run. The averaging over runs or seeds should be documented, as it affects how the small ablation differences should be read.
  • Some claims and references need fixing. The abstract advertises a pre-deployment diagnostic based on partial vs full evidence accuracy, while Appendix D.8 states that such accuracies cannot define a deployment threshold. The conclusion’s claim that the efficiency is “structural rather than a hyperparameter effect” is not tested. Table 1 also has a broken reference (“Table ??”). [1] Liu et al. Pairwise RM: Perform Best-of-N Sampling with Knockout Tournament. arXiv 2025.
Questions for authors
  • On math, what is the distribution of the post-grouping pool size N′, and how often does grouping make Stage B skipped (N′ ≤ 8)? When CAPS selects a different answer from majority voting, how many cases go from incorrect to correct versus correct to incorrect?
  • Could CAPS be evaluated on agentic benchmarks such as SWE-bench or Terminal-Bench, treating full agent trajectories as candidates? What should the partial-evidence view keep (e.g., tool observations, execution errors, final patches), and how often does early elimination discard the only successful trajectory?
1 comment
AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer bqoS,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #52086

Reviewer_tyFgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on best-of-N selection for test-time scaling using pairwise comparison. Given N sampled solutions per query, the goal is to select the most promising one without a large verification budget.

The authors propose CAPS (Cascaded Adaptive Pairwise Selection), a multi-stage selection method in which the generator also acts as the verifier. It reduces verification cost by making fewer comparisons and showing the verifier only partial evidence in the early stage. CAPS first groups solutions by signature (the final answer for math, normalized code text for code). The verifier then runs one elimination round on excerpts: the first and last 50 reasoning words, plus the first 500 code characters or the final answer. The survivors are reduced to at most 4 finalists using full solutions, and a full-solution round-robin among the finalists makes the final selection. An optional rescue step can return one eliminated solution to the finalists.

Across four models and five benchmarks, CAPS exceeds V1-Infer on 14 of 20 settings while using fewer verifier input tokens.

Critical strengths
  1. The method makes sense. Not every solution deserves a full verification budget, and early signals can be useful for pre-filtering before committing to expensive full-solution comparisons.
  2. The method is straightforward, with some interesting design choices, such as using only the first and last parts of the reasoning as partial evidence. This seems reasonable, since the beginning often sets the direction and the end is close to the conclusion. The paper tests different window sizes, but it would be more interesting if the main text also explained why this view works.
  3. The empirical results show higher accuracy than V1-Infer on code at much lower verifier cost. On math, CAPS matches V1-Infer's average accuracy at about half the verifier input tokens.
Critical weaknesses
  1. Simple baselines are missing. A simple choice for best-of-N selection is to select by perplexity (or average log-probability), which requires no verifier calls. Another is to use a smaller model as the pairwise judge. The authors may expect these to perform poorly, but comparing against them is still important for understanding how effective the proposed method is.
  2. The tradeoff with the overall compute is unclear. The main cost comparisons exclude candidate generation and verifier outputs, and the paper notes that its results across pool sizes do not compare equal end-to-end compute. The only end-to-end measurement compares V1-Infer and CAPS on one model and one benchmark. It is therefore unclear whether the same compute would be better spent on selection or on generating more candidates, or how CAPS compares with plain majority voting at equal compute on math.
Questions for authors

Typos and formatting issues:

  1. The Table 1 caption is missing the reference "Table ??".
  2. I assume that in Figure 2, on the y-axis, the "M" in "input tokens (M)" means millions; it would be better to make this clear in the caption.
  3. The baseline unaided sampling (Vanilla) is kind of clear. Please specify this clearly in the experiment section.

Questions:

  1. How often does the excerpt round eliminate all correct candidates, and how often does rescue recover one?
  2. Why use the first 500 code characters, which are often imports and signatures, rather than the end of the code?
1 comment
AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer tyFg,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #52086

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.