acceptodds
Under review as a conference paper at ICLR 2027

When Aggregation Hurts: RESOLVE for Elastic Test-Time Reasoning

Abstract

Scaling test-time compute via structured search—such as Tree-of-Thoughts, Graph-of-Thoughts (GoT), and ensembling—is widely presumed to enhance large language model (LLM) reasoning. In this work, we reveal that expanding search graphs unconditionally often backfires due to Aggregator Poisoning: when candidate traces contain invalid syntax or corrupted logic, presenting them to an LLM synthesizer degrades output quality below single-pass Chain-of-Thought (e.g., on HumanEval, standard GoT collapses from 70.4% to 64.9%). Concurrently, uniform budget expansion wastes compute on straightforward queries, while rigid context-isolation schemes fragment cross-passage dependencies on relational tasks. To resolve these vulnerabilities, we propose RESOLVE (REconciled Synthesis with Optimal-stopping and LLM-gated Verified Exploration). Rather than blind uniform aggregation, RESOLVE maintains full shared context across candidate branches while coordinating: (1)guidance-based width pruning, which syntactically gates out corrupted candidates to shield downstream aggregators from poisoned context, (2)guidance-based depth pruning, dynamically sizing sampling depth via consensus agreement on reasoning or execution feedback on code, and (3)contrastive meta-aggregation, which cross-examines divergent rationales to reconcile conflicting hypotheses. Across 24 benchmark-backbone configurations (, 6 benchmarks, 4 model families), RESOLVE achieves 57.8% overall accuracy compared to 52.1% for GoT, 50.9% for Self-Consistency, 50.4% for ToT, and 50.2% for CoT (outperforming GoT on 20 configurations, tying on 1, and trailing on 3), while reducing token consumption by 40.5% relative to GoT. Under zero-label consensus stopping on natural language and math (4 benchmarks), RESOLVE improves accuracy from 53.4% (GoT) to 56.2% while slashing token costs by 41.1%; on code execution search (2 benchmarks), width gating eliminates aggregator poisoning, elevating accuracy from 49.4% (GoT) to 61.0%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PPd1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies failure modes of structured test-time reasoning, arguing that unconditional aggregation can degrade performance when malformed or logically corrupted candidates enter the synthesizer’s context. It proposes RESOLVE, which combines three components: depth pruning through early stopping, width pruning through syntactic or answer-extractability filters, and contrastive aggregation of the surviving candidates. For natural-language and mathematics tasks, stopping is based on candidate-answer agreement; for code generation, it stops when a candidate passes the benchmark test harness.

The evaluation covers six benchmarks and four open-weight model families. The authors report an average accuracy of 57.8%, compared with 52.1% for Graph-of-Thoughts, while reducing token consumption by 40.5%. The paper additionally presents seeded Qwen3 experiments, pruning ablations, a controlled corruption experiment, and verifier-matched HumanEval comparisons.

Critical strengths
  1. The paper addresses an important practical failure mode of test-time scaling. Fixed candidate budgets can waste computation, and passing malformed outputs directly to a synthesizer is intuitively undesirable. Adaptive stopping and candidate validation are practical ideas for improving inference efficiency.

  2. The empirical scope and reporting effort are substantial. The paper evaluates six benchmarks and four model families while reporting accuracy, tokens, and calls. It also includes seeded experiments, component ablations, verifier-matched controls, and an explicit limitations section that acknowledges uncalibrated consensus and incomplete cost accounting.

Critical weaknesses
  1. The main code-generation results use the evaluation oracle during inference. RESOLVE executes HumanEval candidates against the held-out evaluation tests and LiveCodeBench candidates against decoded private tests, then stops as soon as a candidate passes. These same tests determine the reported final accuracy. The CoT, SC, ToT, and GoT baselines in Table 1 do not receive this information, making the headline coding comparison fundamentally unmatched. Table 10 shows that under verifier-matched conditions, RESOLVE achieves 90.85% versus 90.24% for verifier-matched early stopping, a difference of only one HumanEval problem, while consuming more tokens. Thus, most of the large coding gain appears attributable to oracle-assisted candidate selection rather than the proposed aggregation mechanism. Mixing these test-assisted results into the 57.8% overall average further overstates the general conclusion.

  2. The claimed architecture and supporting evidence are not sufficiently isolated or internally reliable. Appendix D.2 states that RESOLVE uses a merge prompt “identical to GoT,” leaving the proposed contrastive meta-aggregation operationally indistinguishable from the baseline. The claimed Bayesian Value-of-Information procedure reduces in implementation to fixed majority agreement or test passing, without an estimated posterior, expected utility, or cost-sensitive stopping policy. The shared-context claim lacks an isolated-context baseline, while the poisoning experiment injects malformed candidates specifically designed to be removed by the parser. These attribution gaps are compounded by the incorrect GPQA example, conflicting HumanEval numbers and token counts, stale values in Section 4.3, and the use of unseeded runs for the main 24-configuration matrix.

Questions for authors
  1. Were the exact HumanEval held-out tests and LiveCodeBench private tests used both to select candidates during inference and to compute final accuracy? Can the authors provide results using disjoint search-time and evaluation-time tests, with all methods receiving identical verifier access?

  2. What exact operation or prompt distinguishes contrastive meta-aggregation from the merge prompt that Appendix D.2 calls “identical to GoT”? Similarly, where are the claimed Bayesian Value-of-Information calculation and semantic admissibility score implemented?

  3. Can the authors provide a fully audited result ledger explaining the incorrect GPQA example and the conflicting HumanEval accuracies, token counts, losing-configuration values, and call counts across the main text and appendices?

  4. Do the central conclusions remain under paired, seeded evaluation across all four backbones, with confidence intervals and matched controls for early stopping, validity filtering, ordinary GoT merging, context isolation, and naturally occurring rather than injected corrupted candidates?

3 comments

Thank you for a very careful review. It led us to re-audit the implementation and every reported number against the raw per-task logs, and your reading of the code is correct on each point we checked. Below we quote each of your comments and respond to it. We have uploaded a revised PDF in which every change is marked in blue.


Comment 1 (Weakness 1, Question 1).

The main code-generation results use the evaluation oracle during inference. RESOLVE executes HumanEval candidates against the held-out evaluation tests and LiveCodeBench candidates against decoded private tests, then stops as soon as a candidate passes. These same tests determine the reported final accuracy. The CoT, SC, ToT, and GoT baselines in Table 1 do not receive this information, making the headline coding comparison fundamentally unmatched. Table 10 shows that under verifier-matched conditions, RESOLVE achieves 90.85% versus 90.24% for verifier-matched early stopping, a difference of only one HumanEval problem, while consuming more tokens. Thus, most of the large coding gain appears attributable to oracle-assisted candidate selection rather than the proposed aggregation mechanism. Mixing these test-assisted results into the 57.8% overall average further overstates the general conclusion.

Were the exact HumanEval held-out tests and LiveCodeBench private tests used both to select candidates during inference and to compute final accuracy? Can the authors provide results using disjoint search-time and evaluation-time tests, with all methods receiving identical verifier access?

Response. Thank you for this comment. Yes: HumanEval candidates were executed against each problem's held-out test suite, and LiveCodeBench candidates against the same decoded test cases used for scoring, while the baselines had no verifier. As a result, 853 of RESOLVE's 861 correct coding answers (99.1%) are early exits, and the merge produced a passing program on only 8 of the 603 tasks where it ran. We have made the following changes.

  • Overall average. We removed the 57.8% six-task average from the Abstract, Sec. 1, Sec. 4.3 and Table 1, and Table 1 now reports the reasoning and coding regimes separately.
  • Disjoint tests with identical verifier access (new App. H.4, Table 14). Search uses only tests available at inference time: the HumanEval docstring examples (123 of 164 problems) and the LiveCodeBench public tests. Scoring uses only the hidden tests, and LiveCodeBench's private tests are disjoint from the public ones. All methods receive the same four candidates and the same test results. Pass@1, pooled over four backbones:
CoTVM-ESSC+VToT+VGoT+VGoT+V+FRESOLVE+V
HumanEval70.472.472.473.267.471.571.3
LiveCodeBench25.633.033.031.629.533.833.2

With identical verifier access, the coding gap disappears. RESOLVE+V is level with VM-ES (+0.2 pp on LiveCodeBench, 95% CI [0.0, +0.6]; −1.1 pp on HumanEval, [−2.7, +0.6]), and the merge solved 0.5% of the LiveCodeBench tasks that reached it. RESOLVE's coding results thus come from verifier-guided early exit, not from aggregation, as you concluded; Sec. 4.3 and App. G now say so.

  • Table 10 (Table 11 in the revision) now reports three seeds: 91.46 ± 0.50 vs. 90.85 ± 0.50 for VM-ES, exactly one problem apart in each seed.

Comment 2 (Weakness 2, Question 2).

The claimed architecture and supporting evidence are not sufficiently isolated or internally reliable. Appendix D.2 states that RESOLVE uses a merge prompt “identical to GoT,” leaving the proposed contrastive meta-aggregation operationally indistinguishable from the baseline. The claimed Bayesian Value-of-Information procedure reduces in implementation to fixed majority agreement or test passing, without an estimated posterior, expected utility, or cost-sensitive stopping policy. The shared-context claim lacks an isolated-context baseline, while the poisoning experiment injects malformed candidates specifically designed to be removed by the parser. These attribution gaps are compounded by the incorrect GPQA example, conflicting HumanEval numbers and token counts, stale values in Section 4.3, and the use of unseeded runs for the main 24-configuration matrix.

What exact operation or prompt distinguishes contrastive meta-aggregation from the merge prompt that Appendix D.2 calls “identical to GoT”? Similarly, where are the claimed Bayesian Value-of-Information calculation and semantic admissibility score implemented?

Response. Thank you for this comment. Your reading is correct, and the revised Sec. 3, the Fig. 1 caption and Algorithm 1 now describe the operations as implemented.

  • Aggregation. No operation distinguishes it: RESOLVE calls the same merge prompt as our GoT baseline. The difference is only when and on what the merge runs: only when no strict majority forms, and only over admissible candidates. Sec. 1, Sec. 2 and Sec. 3.4 now state this, and we no longer claim a mechanism beyond the GoT merge.
  • Value of information. There is no VoI calculation. Stopping fires when a strict majority of n ≥ 2 extracted answers agree (for code, when a test passes). The posterior and cost language has been removed.
  • Admissibility. The semantic admissibility score is binary answer extractability (an AST parse for code), with no threshold.

The shared-context, poisoning and seeding points are answered under Question 4 in comment 3/3.


Comment 3 (Question 3).

Can the authors provide a fully audited result ledger explaining the incorrect GPQA example and the conflicting HumanEval accuracies, token counts, losing-configuration values, and call counts across the main text and appendices?

Response. Thank you for this request. Every item below was checked against the raw logs.

  1. GPQA example (App. C). It was not a logged trace, and its gold label was wrong (the correct answer is option (B), 4mL²/(3πħ)). It has been removed.
  2. Sec. 2 and the Abstract. "70.4 vs 64.9 (Qwen3, n=164)" are four-backbone averages; the Qwen3 figures are 87.8 vs. 70.7. The label is corrected.
  3. HumanEval/Qwen3. RESOLVE appears as 92.7 (Table 1, unseeded), 90.85 (Table 10, a separate run) and 91.5 ± 0.9 (seeded). Table 10's "CoT" (85.37) is the first sample at temperature 0.7, not greedy decoding (87.8), and its VM-BoN token counts filled in unsampled candidates at the mean length. Captions now state this, and Table 11 reports three seeds.
  4. Sec. 4.3. Its tie and loss values and its call counts (taken from seeded runs) did not match Table 1, and its seeded table had duplicated rows. The section has been rewritten against the revised Table 1.
  5. ES-SC 40.9% (Sec. 5). No reproducible run backs this number. It is withdrawn and replaced by a reproducible early-stop vote on GoT's candidates (32.7% on GPQA; App. H.3).
  6. Issues you did not raise. RESOLVE used larger generation caps on GPQA (2,048 vs. 1,024 tokens) and GSM8K (1,024 vs. 768); Table 1 now reports these cells at the baselines' caps (in blue). Our MATH-500 "GoT" was a decompose-then-solve pipeline, now flagged in App. C.2. On Qwen3 MATH-500, SC returned no extractable answer on 54 of 200 tasks because of truncation in thinking mode.

We will release the per-task logs together with the scripts that compute each table from them.

Comment 4 (Question 4).

Do the central conclusions remain under paired, seeded evaluation across all four backbones, with confidence intervals and matched controls for early stopping, validity filtering, ordinary GoT merging, context isolation, and naturally occurring rather than injected corrupted candidates?

Response. Thank you for this question. Under paired evaluation, part of the conclusions remains and part does not; we report both.

  • Paired evaluation with confidence intervals (App. H.1). At matched generation caps, over all 16 label-free configurations (4 reasoning benchmarks × 4 backbones, 3,192 tasks), RESOLVE scores 54.7% vs. 53.0% for SC (paired difference +1.7 pp, 95% CI [+0.5, +2.9]) and 53.4% for GoT, with 43% fewer tokens than GoT. On the 12 configurations where GoT is the standard k = 4 + merge pipeline (MATH-500 excluded), RESOLVE scores 53.0% vs. 54.0% for GoT (−1.0 pp, [−2.5, +0.5]) with 54% fewer tokens. The intervals come from a paired bootstrap over tasks, stratified by configuration (5,000 resamples). The claim that survives is efficiency at comparable accuracy, not an accuracy gain over GoT.
  • Seeds. Seeded runs exist for Qwen3 (three seeds × six benchmarks; App. A), the HotpotQA ablation (Table 9) and the verifier-matched HumanEval comparison (Table 11). We have no seeded runs for the other three backbones, and our intervals resample tasks, not seeds.
  • Early stopping. Depth pruning alone raises HotpotQA from 53.2 ± 0.5 to 55.2 ± 0.8 at 54% fewer tokens (three seeds; Table 9). It can also hurt: on GPQA, early-stopped queries are correct only 35.1% of the time, because two candidates can agree on a wrong letter (App. H.2).
  • Validity filtering. Its effect is within seed variance: 55.3 ± 0.9 for full RESOLVE vs. 55.2 ± 0.8 without the filter on HotpotQA (Table 9). It never removed a candidate on HumanEval.
  • Ordinary GoT merging (App. H.3). On GoT's own four candidates, merging beats voting on GPQA by 3.9 pp on average and is within 3 pp of it on HotpotQA and GSM8K.
  • Context isolation. Our only isolated-context run is n = 40 on Qwen3 HotpotQA, which is too small to support the claim. Every baseline also gives each candidate the full context, so the revision no longer presents shared context as a distinguishing contribution (Sec. 2.3).
  • Naturally occurring corruption. We agree that the injection study is circular, since the parser removes exactly what we injected. Our main runs did not log candidate traces, so we cannot measure the natural rate. The revision presents aggregator poisoning as a hypothesis motivated by the injection study (Abstract, Sec. 2.1, App. F).

We recognize that these corrections substantially narrow the paper. The revised version claims: (1) on label-free tasks, a gated merge that matches GoT and SC in accuracy at less than half of GoT's tokens; (2) a paired diagnostic of when merging beats voting and when early stopping hurts; and (3) coding results under identical verifier access, where RESOLVE matches verifier-matched early stopping. Thank you again for the thorough review; we would be glad to answer any further questions.

Reviewer_RLorOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RESOLVE, an adaptive test-time reasoning framework that combines early stopping, candidate filtering, and contrastive aggregation to improve LLM reasoning accuracy and efficiency. The method aims to address aggregator poisoning caused by low-quality reasoning candidates while reducing unnecessary computation. Experiments across six benchmarks and four model families demonstrate improvements over existing methods such as CoT, Self-Consistency, ToT, and GoT.

Critical strengths
  1. The paper is clearly written and well-organized, making the proposed approach easy to follow.
  2. The paper investigates an interesting problem of aggregator poisoning in test-time reasoning.
  3. The method achieves improvements in both accuracy and token efficiency across multiple benchmarks and model families.
Critical weaknesses
  1. The proposed approach is very similar to existing works, such as Adaptive-Consistency (Aggarwal et al., 2023), Early-Stopping Self-Consistency (Li et al., 2024), and Graph-of-Thoughts (Besta et al., 2024). The main components, including adaptive early stopping, candidate filtering, and LLM-based aggregation, have already been extensively explored. The proposed framework appears to be an incremental combination of existing techniques rather than a fundamentally new method.
  2. RESOLVE uses unit-test execution feedback to guide early stopping in code generation tasks, while the main baselines do not have access to the same verification signals. This makes it difficult to determine whether the reported improvements come from the proposed framework or the additional verification information. A fair comparison under the same verification conditions is needed.
  3. The ablation results in Table 3 show that adding width pruning to depth pruning only improves accuracy from 74.1% to 74.9%. Furthermore, Table 10 shows that RESOLVE achieves 90.85% accuracy compared with 90.24% for a simpler verifier-matched early-stopping baseline. These relatively small improvements raise concerns about the effectiveness and necessity of the additional components.
  4. Although the paper reports significant token savings, it does not fully account for the overhead of unit-test execution and other inference costs. It remains unclear whether RESOLVE provides substantial efficiency improvements in terms of end-to-end inference latency and computational budget.
Questions for authors
  1. What is the key technical novelty of RESOLVE compared with existing adaptive sampling and aggregation methods, such as Adaptive-Consistency, Early-Stopping Self-Consistency, and GoT?
  2. Could the authors provide more comprehensive comparisons where all methods have access to the same verification signals and computational budget?
  3. Given the marginal improvements in Tables 3 and 10, how much of the overall performance gain can be attributed specifically to width pruning and contrastive meta-aggregation rather than early stopping?
  4. How does RESOLVE compare with existing methods in terms of end-to-end inference latency and computational cost, including the overhead of unit-test execution?
2 comments

Thank you for your thoughtful review, and for finding the paper clearly written and the aggregator-poisoning problem interesting. Below we quote each weakness together with the matching question and respond to it. We have uploaded a revised PDF in which every change is marked in blue.


Comment 1 (Weakness 1, Question 1).

The proposed approach is very similar to existing works, such as Adaptive-Consistency (Aggarwal et al., 2023), Early-Stopping Self-Consistency (Li et al., 2024), and Graph-of-Thoughts (Besta et al., 2024). The main components, including adaptive early stopping, candidate filtering, and LLM-based aggregation, have already been extensively explored. The proposed framework appears to be an incremental combination of existing techniques rather than a fundamentally new method.

What is the key technical novelty of RESOLVE compared with existing adaptive sampling and aggregation methods, such as Adaptive-Consistency, Early-Stopping Self-Consistency, and GoT?

Response. Thank you for this comment. We agree that the individual components are not new. Our stopping rule (stop when a strict majority of n ≥ 2 extracted answers agree) belongs to the Adaptive-Consistency/ESC family, and our synthesis step uses the same merge prompt as our GoT baseline. The submission's language ("Bayesian VoI", "contrastive cross-examination") overstated both. The revised Sec. 3 states the operations as implemented, and Sec. 4.5 credits the stopping rule to Adaptive-Consistency and ESC.

The contribution is the control flow that combines them. ESC-style methods stop early and then vote, and GoT always merges all k candidates. RESOLVE stops on agreement and merges only on disagreement, over the admissible candidates. This combination matters in both directions:

  • Against stop-and-vote: applying the ESC rule with a vote to GoT's own candidates gives 32.7% on GPQA, vs. 34.0% for RESOLVE and 36.7% for GoT (four-backbone means; App. H.3). On identical candidates, merging beats voting on GPQA by 3.9 pp on average.
  • Against always-merge: over all 16 label-free configurations at matched caps, RESOLVE scores 54.7% vs. 53.4% for GoT and 53.0% for SC (paired difference to SC +1.7 pp, 95% CI [+0.5, +2.9]) with 43% fewer tokens than GoT (App. H.1).

We now present RESOLVE as an efficient composition of known components, not as a fundamentally new technique, and the Abstract and Sec. 1 were revised accordingly.


Comment 2 (Weakness 2, Question 2).

RESOLVE uses unit-test execution feedback to guide early stopping in code generation tasks, while the main baselines do not have access to the same verification signals. This makes it difficult to determine whether the reported improvements come from the proposed framework or the additional verification information. A fair comparison under the same verification conditions is needed.

Could the authors provide more comprehensive comparisons where all methods have access to the same verification signals and computational budget?

Response. Thank you for this comment. You are right: in Table 1, RESOLVE's early exit executed the benchmarks' evaluation tests, and no baseline had a verifier. The new App. H.4 (Table 14) gives every method the same verifier, built only from tests available at inference time (HumanEval docstring examples; LiveCodeBench public tests), scores on the hidden tests, and gives all methods the same four candidates per problem. Pooled over four backbones (pass@1):

CoTVM-ESSC+VToT+VGoT+VGoT+V+FRESOLVE+V
HumanEval70.472.472.473.267.471.571.3
LiveCodeBench25.633.033.031.629.533.833.2

Under the same verifier, the verified methods lie within about 2 pp, and RESOLVE+V is level with verifier-matched early stopping (VM-ES): +0.2 pp on LiveCodeBench (95% CI [0.0, +0.6]) and −1.1 pp on HumanEval ([−2.7, +0.6]). The coding gains in Table 1 therefore come from verifier access, not from our framework; Sec. 4.3 now says so.

For the computational budget, every multi-candidate method uses k = 4. Our audit also found that RESOLVE had used a larger generation cap on GPQA (2,048 vs. 1,024 tokens) and GSM8K (1,024 vs. 768). Table 1 now reports these cells at the baselines' caps (in blue), so all methods share the same cap on every benchmark.

Comment 3 (Weakness 3, Question 3).

The ablation results in Table 3 show that adding width pruning to depth pruning only improves accuracy from 74.1% to 74.9%. Furthermore, Table 10 shows that RESOLVE achieves 90.85% accuracy compared with 90.24% for a simpler verifier-matched early-stopping baseline. These relatively small improvements raise concerns about the effectiveness and necessity of the additional components.

Given the marginal improvements in Tables 3 and 10, how much of the overall performance gain can be attributed specifically to width pruning and contrastive meta-aggregation rather than early stopping?

Response. Thank you for this comment. Your concern is warranted: almost all of the gain comes from early stopping.

  • Width pruning. We reran the ablation on HotpotQA with three seeds (App. E, Table 9). Width pruning alone scores 54.0 ± 0.7 vs. 53.2 ± 0.5 with neither component, and full RESOLVE scores 55.3 ± 0.9 vs. 55.2 ± 0.8 without width pruning, both within seed variance. The filter removed a candidate on only 3–6% of HotpotQA tasks. On HumanEval it removed no candidate on any of the 164 tasks, so the HumanEval width-pruning gaps in Table 3 are run-to-run variance.
  • Aggregation. Your Table 10 (Table 11 in the revision) now has three seeds: RESOLVE 91.46 ± 0.50 vs. VM-ES 90.85 ± 0.50, a margin of exactly one problem in each seed. Under the independent public-test verifier (App. H.4), the merge solved 0.5% of the LiveCodeBench tasks that reached it.
  • Early stopping. Depth pruning alone raises HotpotQA from 53.2 ± 0.5 to 55.2 ± 0.8 while using 54% fewer tokens (Table 9).

The merge contributes only where candidates disagree and no verifier exists: on GPQA, it beats voting on identical candidates by 3.9 pp on average (App. H.3). Sec. 4.5 now attributes RESOLVE's gains to depth pruning, and App. H.2 reports how often each operation fires.


Comment 4 (Weakness 4, Question 4).

Although the paper reports significant token savings, it does not fully account for the overhead of unit-test execution and other inference costs. It remains unclear whether RESOLVE provides substantial efficiency improvements in terms of end-to-end inference latency and computational budget.

How does RESOLVE compare with existing methods in terms of end-to-end inference latency and computational cost, including the overhead of unit-test execution?

Response. Thank you for this comment. We now report latency and test-execution time (App. H.1 and H.4).

  • Generation latency. Over the 12 configurations at matched caps where GoT is the standard pipeline, mean summed generation time per task is 88 s for RESOLVE, 135 s for ToT, 148 s for SC, 157 s for GoT and 34 s for CoT. All methods ran sequentially and unbatched on the same GPU type, so these figures are indicative.
  • Unit-test execution. In the verifier-matched code runs, test execution adds 0.4 s (HumanEval) and 3.4 s (LiveCodeBench) per task for RESOLVE+V, against 33 s and 71 s of generation time; no method exceeds 4.4 s per task. Test execution is therefore a small fraction of end-to-end cost.

We also added a structural limitation (Sec. 6). RESOLVE samples sequentially, so on a batched serving stack SC and GoT can generate their k samples in parallel while RESOLVE cannot, and its wall-clock advantage may shrink there.


Thank you again for your review. We hope these results clarify where RESOLVE helps and where it does not, and we would be glad to answer any further questions.

Reviewer_f7kVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper pays attention to two problems in structured test-time reasoning: (1) unreliable reasoning paths can hurt the final answer when combined (aggregator poisoning), and (2) fixed sampling budgets often waste computation. The authors propose RESOLVE to address both issues. RESOLVE uses three main techniques: depth pruning to stop sampling when enough evidence is available, width pruning to filter out unreliable candidates, and contrastive meta-aggregation to combine different reasoning paths into a final answer. It also shares context across reasoning branches to support more complex reasoning tasks. The authors evaluate RESOLVE on six benchmarks using four LLM backbones. Compared with Graph-of-Thoughts, RESOLVE improves average accuracy from 52.1% to 57.8% while reducing token usage by 40.5%.

Critical strengths
  1. Clear motivation. The paper identifies aggregator poisoning, an interesting failure mode where unreliable reasoning traces can hurt the final prediction.

  2. Practical and efficient framework. RESOLVE combines early stopping and candidate filtering to reduce unnecessary computation without additional training.

  3. Broad experimental evaluation. Experiments across six benchmarks and four LLM backbones show improvements in accuracy and token efficiency, supported by ablation and robustness studies.

Critical weaknesses
  1. The comparison among different methods may be unfair. On HumanEval and LiveCodeBench, RESOLVE uses execution feedback from test cases to enable early stopping, whereas the main comparisons with CoT, SC, ToT, and GoT do not provide these baselines with equivalent verifier-in-the-loop feedback. More importantly, RESOLVE appears to use the evaluation test suite itself, rather than an independent set of publicly available example tests.

  2. Verifier-matched comparisons substantially weaken the claimed performance gains. From Appendix I (Table 10), RESOLVE achieves only a 0.61 percentage-point improvement over the verifier-matched early-stopping baseline. These results suggest that much of the reported performance improvement on code generation tasks may come from execution-based verification and candidate selection, rather than the proposed Contrastive Meta-Aggregation mechanism.

Questions for authors

Please see Weaknesses 1- 2. And I also think the following point influences my evaluation:

  1. Poor visual quality of Figure 1. Figure 1 appears to be a rasterized image rather than a vector graphic, and its overall visual quality is relatively poor. In addition, the top and bottom parts are not connected well in terms of semantics.
1 comment

Thank you for your careful review, and for recognizing the clear motivation (aggregator poisoning), the practical, training-free design and the breadth of the evaluation across six benchmarks and four backbones. Below we quote each of your comments and respond to it. Changes in the revised PDF are marked in blue.


Weakness 1.

The comparison among different methods may be unfair. On HumanEval and LiveCodeBench, RESOLVE uses execution feedback from test cases to enable early stopping, whereas the main comparisons with CoT, SC, ToT, and GoT do not provide these baselines with equivalent verifier-in-the-loop feedback. More importantly, RESOLVE appears to use the evaluation test suite itself, rather than an independent set of publicly available example tests.

Response. Thank you for your comment. Following your suggestion, we added a comparison in which every method receives the same verifier, built from independent, publicly available example tests (new App. J, Table 11). RESOLVE keeps a clear gain over CoT and GoT in this setting, at a much lower cost.

  • Setup. The verifier uses only tests available at inference time: the docstring examples on HumanEval and the public tests on LiveCodeBench. Scoring uses the hidden tests, which on LiveCodeBench are disjoint from the public ones by construction. All multi-candidate methods share the same four candidates and the same test results per problem.
  • Results. Pass@1, pooled over the four backbones:
CoTVM-ESSC+VToT+VGoT+VGoT+V+FRESOLVE+V
HumanEval70.472.472.473.267.471.571.3
LiveCodeBench25.633.033.031.629.533.833.2
  • Gains under a shared verifier.
    • Over CoT: +0.9 pp on HumanEval and +7.6 pp on LiveCodeBench.
    • Over GoT+V, which receives the same test feedback: +4.0 pp on HumanEval (95% CI [+2.1, +5.8]) and +3.8 pp on LiveCodeBench ([+2.1, +5.4]), with 49% and 35% fewer tokens.
    • RESOLVE+V is competitive with the other verified methods (within about 1 pp of VM-ES, SC+V and GoT+V+F), at a fraction of the cost of the aggregation pipelines.

Regarding the test suite in Table 1: RESOLVE's early exit used each benchmark's own tests, which the baselines did not receive. Sec. 4.3 and the Table 1 caption now state this explicitly, and App. J provides the matched comparison above.

Weakness 2.

Verifier-matched comparisons substantially weaken the claimed performance gains. From Appendix I (Table 10), RESOLVE achieves only a 0.61 percentage-point improvement over the verifier-matched early-stopping baseline. These results suggest that much of the reported performance improvement on code generation tasks may come from execution-based verification and candidate selection, rather than the proposed Contrastive Meta-Aggregation mechanism.

Response. Thank you for your comment. We extended Table 10 (App. I) to three seeds, and the result is consistent: RESOLVE has the highest accuracy in every seed (91.46 ± 0.50 vs. 90.85 ± 0.50 for VM-ES) and stays far above GoT without a verifier (72.15 ± 2.30).

RESOLVE's overall gain also does not depend on a verifier. On the reasoning benchmarks, where stopping is label-free, we compared all methods at identical generation settings: over the 16 configurations (HotpotQA, GPQA, MATH-500 and GSM8K × 4 backbones), RESOLVE scores 54.7% vs. 53.0% for SC (paired difference +1.7 pp, 95% CI [+0.5, +2.9]), while using 43% fewer tokens than GoT.

We agree that on code generation the verifier accounts for most of the gain, and Sec. 4.3 now attributes the coding results accordingly. Sec. 3.3 also clarifies that the synthesis step uses the same merge prompt as GoT. RESOLVE's contribution lies in deciding when to merge (only when no majority or passing program emerges) and what to merge (only admissible candidates). This is what yields its accuracy gain over always-merging with the same verifier (GoT+V, Weakness 1) and its token savings over GoT.


Question (Figure 1).

Poor visual quality of Figure 1. Figure 1 appears to be a rasterized image rather than a vector graphic, and its overall visual quality is relatively poor. In addition, the top and bottom parts are not connected well in terms of semantics.

Response. Thank you for your comment. We redrew Figure 1 (p. 4) so that its two halves describe the same query. The top row shows the control flow, and numbered panels below expand each step on a single running example (find all x with |x − 1| = 2). The example flows from depth pruning (panel 1) through the admissibility gate (panel 2) into synthesis (panel 3). The caption was rewritten to match the implemented operations.


Thank you again for your review. We hope the verifier-matched results and the revised figure address your concerns, and we would be glad to answer any further questions.

Comments on the paper

1 comment

Dear Reviewers, thank you once again for your constructive feedback. With the discussion period nearing its end, we would greatly appreciate knowing whether our rebuttal and new experimental results have resolved your concerns. We remain available to clarify any remaining questions before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.