- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies failure modes of structured test-time reasoning, arguing that unconditional aggregation can degrade performance when malformed or logically corrupted candidates enter the synthesizer’s context. It proposes RESOLVE, which combines three components: depth pruning through early stopping, width pruning through syntactic or answer-extractability filters, and contrastive aggregation of the surviving candidates. For natural-language and mathematics tasks, stopping is based on candidate-answer agreement; for code generation, it stops when a candidate passes the benchmark test harness.
The evaluation covers six benchmarks and four open-weight model families. The authors report an average accuracy of 57.8%, compared with 52.1% for Graph-of-Thoughts, while reducing token consumption by 40.5%. The paper additionally presents seeded Qwen3 experiments, pruning ablations, a controlled corruption experiment, and verifier-matched HumanEval comparisons.
Critical strengths
-
The paper addresses an important practical failure mode of test-time scaling. Fixed candidate budgets can waste computation, and passing malformed outputs directly to a synthesizer is intuitively undesirable. Adaptive stopping and candidate validation are practical ideas for improving inference efficiency.
-
The empirical scope and reporting effort are substantial. The paper evaluates six benchmarks and four model families while reporting accuracy, tokens, and calls. It also includes seeded experiments, component ablations, verifier-matched controls, and an explicit limitations section that acknowledges uncalibrated consensus and incomplete cost accounting.
Critical weaknesses
-
The main code-generation results use the evaluation oracle during inference. RESOLVE executes HumanEval candidates against the held-out evaluation tests and LiveCodeBench candidates against decoded private tests, then stops as soon as a candidate passes. These same tests determine the reported final accuracy. The CoT, SC, ToT, and GoT baselines in Table 1 do not receive this information, making the headline coding comparison fundamentally unmatched. Table 10 shows that under verifier-matched conditions, RESOLVE achieves 90.85% versus 90.24% for verifier-matched early stopping, a difference of only one HumanEval problem, while consuming more tokens. Thus, most of the large coding gain appears attributable to oracle-assisted candidate selection rather than the proposed aggregation mechanism. Mixing these test-assisted results into the 57.8% overall average further overstates the general conclusion.
-
The claimed architecture and supporting evidence are not sufficiently isolated or internally reliable. Appendix D.2 states that RESOLVE uses a merge prompt “identical to GoT,” leaving the proposed contrastive meta-aggregation operationally indistinguishable from the baseline. The claimed Bayesian Value-of-Information procedure reduces in implementation to fixed majority agreement or test passing, without an estimated posterior, expected utility, or cost-sensitive stopping policy. The shared-context claim lacks an isolated-context baseline, while the poisoning experiment injects malformed candidates specifically designed to be removed by the parser. These attribution gaps are compounded by the incorrect GPQA example, conflicting HumanEval numbers and token counts, stale values in Section 4.3, and the use of unseeded runs for the main 24-configuration matrix.
Questions for authors
-
Were the exact HumanEval held-out tests and LiveCodeBench private tests used both to select candidates during inference and to compute final accuracy? Can the authors provide results using disjoint search-time and evaluation-time tests, with all methods receiving identical verifier access?
-
What exact operation or prompt distinguishes contrastive meta-aggregation from the merge prompt that Appendix D.2 calls “identical to GoT”? Similarly, where are the claimed Bayesian Value-of-Information calculation and semantic admissibility score implemented?
-
Can the authors provide a fully audited result ledger explaining the incorrect GPQA example and the conflicting HumanEval accuracies, token counts, losing-configuration values, and call counts across the main text and appendices?
-
Do the central conclusions remain under paired, seeded evaluation across all four backbones, with confidence intervals and matched controls for early stopping, validity filtering, ordinary GoT merging, context isolation, and naturally occurring rather than injected corrupted candidates?
3 comments
Comment 2 (Weakness 2, Question 2).
The claimed architecture and supporting evidence are not sufficiently isolated or internally reliable. Appendix D.2 states that RESOLVE uses a merge prompt “identical to GoT,” leaving the proposed contrastive meta-aggregation operationally indistinguishable from the baseline. The claimed Bayesian Value-of-Information procedure reduces in implementation to fixed majority agreement or test passing, without an estimated posterior, expected utility, or cost-sensitive stopping policy. The shared-context claim lacks an isolated-context baseline, while the poisoning experiment injects malformed candidates specifically designed to be removed by the parser. These attribution gaps are compounded by the incorrect GPQA example, conflicting HumanEval numbers and token counts, stale values in Section 4.3, and the use of unseeded runs for the main 24-configuration matrix.
What exact operation or prompt distinguishes contrastive meta-aggregation from the merge prompt that Appendix D.2 calls “identical to GoT”? Similarly, where are the claimed Bayesian Value-of-Information calculation and semantic admissibility score implemented?
Response. Thank you for this comment. Your reading is correct, and the revised Sec. 3, the Fig. 1 caption and Algorithm 1 now describe the operations as implemented.
- Aggregation. No operation distinguishes it: RESOLVE calls the same merge prompt as our GoT baseline. The difference is only when and on what the merge runs: only when no strict majority forms, and only over admissible candidates. Sec. 1, Sec. 2 and Sec. 3.4 now state this, and we no longer claim a mechanism beyond the GoT merge.
- Value of information. There is no VoI calculation. Stopping fires when a strict majority of n ≥ 2 extracted answers agree (for code, when a test passes). The posterior and cost language has been removed.
- Admissibility. The semantic admissibility score is binary answer extractability (an AST parse for code), with no threshold.
The shared-context, poisoning and seeding points are answered under Question 4 in comment 3/3.
Comment 3 (Question 3).
Can the authors provide a fully audited result ledger explaining the incorrect GPQA example and the conflicting HumanEval accuracies, token counts, losing-configuration values, and call counts across the main text and appendices?
Response. Thank you for this request. Every item below was checked against the raw logs.
- GPQA example (App. C). It was not a logged trace, and its gold label was wrong (the correct answer is option (B), 4mL²/(3πħ)). It has been removed.
- Sec. 2 and the Abstract. "70.4 vs 64.9 (Qwen3, n=164)" are four-backbone averages; the Qwen3 figures are 87.8 vs. 70.7. The label is corrected.
- HumanEval/Qwen3. RESOLVE appears as 92.7 (Table 1, unseeded), 90.85 (Table 10, a separate run) and 91.5 ± 0.9 (seeded). Table 10's "CoT" (85.37) is the first sample at temperature 0.7, not greedy decoding (87.8), and its VM-BoN token counts filled in unsampled candidates at the mean length. Captions now state this, and Table 11 reports three seeds.
- Sec. 4.3. Its tie and loss values and its call counts (taken from seeded runs) did not match Table 1, and its seeded table had duplicated rows. The section has been rewritten against the revised Table 1.
- ES-SC 40.9% (Sec. 5). No reproducible run backs this number. It is withdrawn and replaced by a reproducible early-stop vote on GoT's candidates (32.7% on GPQA; App. H.3).
- Issues you did not raise. RESOLVE used larger generation caps on GPQA (2,048 vs. 1,024 tokens) and GSM8K (1,024 vs. 768); Table 1 now reports these cells at the baselines' caps (in blue). Our MATH-500 "GoT" was a decompose-then-solve pipeline, now flagged in App. C.2. On Qwen3 MATH-500, SC returned no extractable answer on 54 of 200 tasks because of truncation in thinking mode.
We will release the per-task logs together with the scripts that compute each table from them.
Comment 4 (Question 4).
Do the central conclusions remain under paired, seeded evaluation across all four backbones, with confidence intervals and matched controls for early stopping, validity filtering, ordinary GoT merging, context isolation, and naturally occurring rather than injected corrupted candidates?
Response. Thank you for this question. Under paired evaluation, part of the conclusions remains and part does not; we report both.
- Paired evaluation with confidence intervals (App. H.1). At matched generation caps, over all 16 label-free configurations (4 reasoning benchmarks × 4 backbones, 3,192 tasks), RESOLVE scores 54.7% vs. 53.0% for SC (paired difference +1.7 pp, 95% CI [+0.5, +2.9]) and 53.4% for GoT, with 43% fewer tokens than GoT. On the 12 configurations where GoT is the standard k = 4 + merge pipeline (MATH-500 excluded), RESOLVE scores 53.0% vs. 54.0% for GoT (−1.0 pp, [−2.5, +0.5]) with 54% fewer tokens. The intervals come from a paired bootstrap over tasks, stratified by configuration (5,000 resamples). The claim that survives is efficiency at comparable accuracy, not an accuracy gain over GoT.
- Seeds. Seeded runs exist for Qwen3 (three seeds × six benchmarks; App. A), the HotpotQA ablation (Table 9) and the verifier-matched HumanEval comparison (Table 11). We have no seeded runs for the other three backbones, and our intervals resample tasks, not seeds.
- Early stopping. Depth pruning alone raises HotpotQA from 53.2 ± 0.5 to 55.2 ± 0.8 at 54% fewer tokens (three seeds; Table 9). It can also hurt: on GPQA, early-stopped queries are correct only 35.1% of the time, because two candidates can agree on a wrong letter (App. H.2).
- Validity filtering. Its effect is within seed variance: 55.3 ± 0.9 for full RESOLVE vs. 55.2 ± 0.8 without the filter on HotpotQA (Table 9). It never removed a candidate on HumanEval.
- Ordinary GoT merging (App. H.3). On GoT's own four candidates, merging beats voting on GPQA by 3.9 pp on average and is within 3 pp of it on HotpotQA and GSM8K.
- Context isolation. Our only isolated-context run is n = 40 on Qwen3 HotpotQA, which is too small to support the claim. Every baseline also gives each candidate the full context, so the revision no longer presents shared context as a distinguishing contribution (Sec. 2.3).
- Naturally occurring corruption. We agree that the injection study is circular, since the parser removes exactly what we injected. Our main runs did not log candidate traces, so we cannot measure the natural rate. The revision presents aggregator poisoning as a hypothesis motivated by the injection study (Abstract, Sec. 2.1, App. F).
We recognize that these corrections substantially narrow the paper. The revised version claims: (1) on label-free tasks, a gated merge that matches GoT and SC in accuracy at less than half of GoT's tokens; (2) a paired diagnostic of when merging beats voting and when early stopping hurts; and (3) coding results under identical verifier access, where RESOLVE matches verifier-matched early stopping. Thank you again for the thorough review; we would be glad to answer any further questions.
Thank you for a very careful review. It led us to re-audit the implementation and every reported number against the raw per-task logs, and your reading of the code is correct on each point we checked. Below we quote each of your comments and respond to it. We have uploaded a revised PDF in which every change is marked in blue.
Comment 1 (Weakness 1, Question 1).
Response. Thank you for this comment. Yes: HumanEval candidates were executed against each problem's held-out test suite, and LiveCodeBench candidates against the same decoded test cases used for scoring, while the baselines had no verifier. As a result, 853 of RESOLVE's 861 correct coding answers (99.1%) are early exits, and the merge produced a passing program on only 8 of the 603 tasks where it ran. We have made the following changes.
With identical verifier access, the coding gap disappears. RESOLVE+V is level with VM-ES (+0.2 pp on LiveCodeBench, 95% CI [0.0, +0.6]; −1.1 pp on HumanEval, [−2.7, +0.6]), and the merge solved 0.5% of the LiveCodeBench tasks that reached it. RESOLVE's coding results thus come from verifier-guided early exit, not from aggregation, as you concluded; Sec. 4.3 and App. G now say so.