acceptodds
Under review as a conference paper at ICLR 2027

OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation

Abstract

Test-time compute scaling is a primary axis for improving LLM reasoning. Existing methods primarily scale depth by extending a single reasoning trace. Scaling breadth by sampling multiple candidates in parallel is straightforward, but introduces a selection bottleneck: choosing the best candidate without a ground-truth verifier, since pointwise LLM judging is noisy and biased. To address this, we introduce OpenDeepThink, a population-based test-time compute framework that selects via pairwise Bradley–Terry comparison. Each generation, the LLM judges random pairs of candidates and aggregates votes via Bradley–Terry into a global ranking; top-ranked candidates are preserved and the top three quarters are mutated using the natural-language critiques produced during comparison; the bottom quarter is discarded. OpenDeepThink raises Gemini 3.1 Pro's effective Codeforces Elo by +405 points in eight sequential LLM-call rounds ( 27 minutes wall-clock). The pipeline transfers across weaker and stronger models without retuning, and on the multi-domain HLE benchmark, gains appear concentrated in objectively verifiable domains and reverse in subjective ones. We release CF-73, a curated set of 73 expert-rated Codeforces problems with International Grandmaster annotation and 99% local-evaluation agreement against the official verdict.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time compute scaling, parallel reasoning, Bradley-Terry model, LLM-as-a-judge, evolutionary search, competitive programming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_VvqiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents OpenDeepThink, a framework for scaling LLM reasoning through parallel sampling without an external correctness verifier. The same model generates and compares candidate solutions, using Bradley–Terry rankings for selection and comparison critiques for iterative refinement. Experiments primarily focus on competitive programming, showing improvements over the base model and the implemented baselines.

Critical strengths
  1. The paper identifies candidate selection as a key challenge in parallel reasoning without an external verifier. It explains how pairwise comparison, ranking, and feedback-guided refinement fit together, and Algorithm 1 makes the overall procedure straightforward to understand.
  2. Beyond the main results on two programming benchmarks and HLE, the experiments examine how performance changes across generations and how different feedback strategies affect refinement.
  3. The same hyperparameters are used for Gemini 3.1 Pro, Gemini 3 Flash, and Gemini 2.5 Pro, with improvements in both candidate quality and final selection accuracy, suggesting that iterative refinement adds value beyond selecting from the initial samples.
Critical weaknesses
  1. The baseline comparisons are not strong enough. The related work discusses several closely related test-time scaling methods, including Duel-Evolve and V1, but neither is evaluated directly. Instead, the comparisons focus on pointwise selection and a simplified Self-Refine implementation. These results show improvements over relatively simple alternatives, but do not establish what the proposed pipeline offers over the closest existing methods.
  2. The paper reports API-call counts, a latency estimate, and a BT-only scaling study, but lacks an end-to-end comparison of accuracy against token usage and measured wall-clock time across methods. Matching call counts does not necessarily match cost, since generation, comparison, and refinement have different token requirements. Given that efficiency is a central motivation, better accuracy than simple baselines alone is not enough to show that this is a better use of test-time compute.
  3. The pipeline requires a population size, comparison budgets, an iteration count, and an elite ratio. While some settings are examined, the full pipeline lacks a systematic sensitivity analysis, and the elite ratio was chosen through informal tuning. This leaves practical questions about how to allocate a new budget or adapt the method to a new task without costly trial and error.
Questions for authors
  1. Could you expand the comparisons to include stronger, closely related test-time scaling baselines? The current pointwise and modified Self-Refine baselines do not establish the pipeline’s advantage over existing approaches discussed in §2.

  2. With these additional baselines included, how does your method compare in accuracy versus total token usage and measured end-to-end latency under comparable concurrency limits? Such comparisons would help establish whether the framework offers a better cost-performance trade-off, rather than simply higher accuracy.

  3. How sensitive is the full pipeline to population size, comparison budgets, iteration count, and elite ratio? What guidance can you offer for choosing these settings, and deciding when to stop, on a new task without access to correctness labels?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Vvqi

Thank you for the practical questions. Numbers below use the corrected data (see General Response).

Q1. Could you expand the comparisons to include stronger, closely related test-time scaling baselines?

Thank you for the suggestion. We compared with V1-Infer (public code), used as our final selector on the same pools with the same judge model. It performs on par with BT (74.4% vs 71.8% on 39 mixed Hard pools; the CI includes zero), but needs 6 sequential batches instead of one. Duel-Evolve has no public code, needs 150–200 sequential generations, and uses public-test execution on code, which our setting excludes.

Q2. How does your method compare in accuracy versus total token usage and measured end-to-end latency under comparable concurrency limits?

This is an important point, since efficiency is a central motivation of the paper. With thinking tokens logged (over 90% of cost), on Hard:

MethodCost per problemTop-1
OpenDeepThink~103 USD50.0%
Self-Refine + pointwise~96 USD42.8%
Self-Refine + BT~85 USD51.6%

Costs are of the same order, and pairwise selection is what lifts accuracy. The ideal wall-clock time is about 28 minutes (8 rounds); in practice, time was limited by API throughput, not depth.

Q3. How sensitive is the full pipeline to population size, comparison budgets, iteration count, and elite ratio? … and deciding when to stop?

Thank you for asking; we will add concrete guidance to the paper.

  • Comparison budget: during evolution, each candidate gets K = 4 ≈ log2 n comparisons. In the final round, more comparisons never hurt, but the gain beyond about 20 is small (within about 2 points of 100 overall, about 3 on Hard), so this is the first place to cut under a tight budget.
  • Population: larger pools give selection more to choose from, with gains leveling off beyond about 15 candidates.
  • Iterations: most rescues happen in the first generation, so later generations have diminishing returns.
  • Elites: each generation copies the top quarter unchanged, so good candidates cannot be broken by mutation. An earlier code version without this copy lost some correct elites to mutation; with it, none were lost.
  • Stopping without labels: stop once at least 75% of a round's verdicts are ties. This saves 46% of calls with no accuracy loss, and rarely fires on hard problems, where the budget is needed.

Recipe for a new task. Default: n = 20, K = 4, T = 3, 10 final rounds (about 285 calls). Under a tight budget, a lighter setting such as the one we used on HLE (n = 12, T = 2, 6 final rounds; about 115 calls) applies the reductions above together; we validated each reduction on programming, but not this exact combination. In both cases, stop early at 75% ties. We will add this guidance to the paper.

Thank you for the detailed response and the additional experiments. I appreciate the effort, but my main concern remains. The V1-Infer result compares only the final selector on the same candidate pools, so it does not show how the full OpenDeepThink pipeline compares with other test-time scaling methods. I would also like to see these methods compared end to end under the same setup, especially in terms of measured wall-clock time, token usage, and accuracy. This is important because parallel efficiency is a key motivation of the paper. The additional tuning advice is helpful, though the number of manually chosen hyperparameters may still make the method difficult to use on new tasks.

Thank you for the reply.

The V1-Infer result compares only the final selector on the same candidate pools … compared end to end under the same setup, especially in terms of measured wall-clock time, token usage, and accuracy.

We agree. We ran the full V1 pipeline (sample 20 solutions, then the V1 tournament) on the same initial samples. Results on the 62 Hard problems with Gemini 3.1 Pro:

MethodCallsCost per problemTop-1
Best-of-20 + V1-Infer (V1 pipeline)80~28 USD29.0%
Self-Refine + pointwise300~96 USD42.8%
Self-Refine + pairwise BT240~85 USD51.6%
OpenDeepThink285~103 USD50.0%
OpenDeepThink, V1-Infer as final selector245~89 USD51.6%

The V1 pipeline is cheaper and faster (about 38 vs. 75 minutes per problem, measured from call logs), but it can only select from the initial samples, and half of the Hard problems have no correct initial sample. OpenDeepThink is 21.0 points higher (95% CI [11.3, 30.6]). We also did a live run with DeepSeek V4 Flash on 5 problems, sending all calls of a round at once: OpenDeepThink took 61 minutes and Self-Refine took 72 minutes. The 28 minutes in our first reply was only a lower bound, and we corrected it in the revised PDF.

the number of manually chosen hyperparameters may still make the method difficult to use on new tasks

We used the same default setting for all models (three Gemini models, DeepSeek V4 Flash and Qwen3.5-9B) without any retuning, so we expect the defaults to be a reasonable starting point for a new task. Also, the label-free stopping rule from our first reply can end a run early on easy problems and save calls.

Thank you again for the discussion. We hope the end-to-end comparison addresses your main concern, and we are happy to clarify anything before the discussion ends.

Reviewer_MH4GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OpenDeepThink, a population-based test-time scaling framework combining parallel candidate generation, pairwise LLM judging, Bradley–Terry aggregation, and feedback-driven evolutionary refinement. The method achieves substantial improvements on competitive programming benchmarks, including a reported +405 Codeforces Elo gain for Gemini 3.1 Pro. The authors also evaluate cross-model and cross-domain transfer and release CF-73, a curated benchmark of 73 competitive programming problems.

Critical strengths

S1. Strong empirical results.

The method substantially improves competitive programming performance, particularly on difficult problems, where accuracy increases from 11% to 50%.

S2. Comprehensive ablation studies.

The paper investigates the contributions of pairwise selection, feedback-driven mutation, and inference budget allocation, providing useful insights into the effectiveness of different components.

Critical weaknesses

W1. Limited methodological novelty.

The proposed framework closely resembles existing verifier-free evolutionary inference methods, particularly Duel-Evolve (Karlekar et al., 2026), which already combines pairwise LLM preferences, Bradley–Terry aggregation, and evolutionary optimization. V1 (Singh et al., 2026) also establishes the benefits of pairwise self-verification. The main distinction appears to be parallel execution and critique-guided mutation rather than a fundamentally new algorithm.

W2. Unclear contribution of Bradley–Terry aggregation.

Although BT aggregation is emphasized as a central contribution, Appendix A.7 shows that BT and the simpler Borda count achieve identical accuracy on the main competitive programming benchmark (82.8%). This raises questions about whether BT is necessary and whether the observed gains primarily come from evolutionary refinement and additional computation.

W3. Limited cross-model generalization.

The experiments primarily focus on Gemini-family models, while the only non-Gemini evaluation covers five Codeforces problems using DeepSeek V4 Flash (Appendix A.7). This limited evidence is insufficient to establish the claimed generalizability across model families.

Questions for authors

Q1. What is the fundamental methodological difference between OpenDeepThink and Duel-Evolve, beyond parallelization and the use of natural-language critiques?

Q2. Given that BT and Borda count achieve identical performance, what evidence supports the necessity of Bradley–Terry aggregation?

Q3. How does OpenDeepThink compare with Duel-Evolve, V1, and Squeeze Evolve under matched inference budgets?

Q4. Can the authors provide more extensive evaluations on non-Gemini models and explain the performance degradation on HLE?

3 comments
AuthorsOpenReview ↗

Response to Reviewer MH4G

Thank you for the review. We agree that pairwise preferences, BT and evolutionary search each appear in prior work, and we revise the paper to say so.

Q1. What is the fundamental methodological difference between OpenDeepThink and Duel-Evolve, beyond parallelization and the use of natural-language critiques?

Thank you for this important question. Three differences go beyond parallelism. (1) Depth. Duel-Evolve runs 150–200 adaptive generations; we use 8 sequential rounds in total. We study how to turn parallel compute into accuracy under bounded depth, which rules out adaptive duel allocation. (2) No tests. On code, Duel-Evolve shows public-test execution traces to its generator and judge; we execute no tests at all. (3) Population and feedback. Duel-Evolve grows a pool with confidence-based pruning and conditions on posterior scores; we keep a fixed population with elites and mutate with full pairwise critiques, which significantly raise the rescue rate (+4.5 points). Both constraints reflect practical use: new tasks often come without tests, and wall-clock time is set by depth.

Q2. Given that BT and Borda count achieve identical performance, what evidence supports the necessity of Bradley–Terry aggregation?

Thank you for raising this point. BT is not necessary for the gains: BT and Borda perform similarly, and both beat raw win rate when comparisons are sparse. The gains come from evolution (on Hard, correct candidates rise from 9.7% to 40.8%) and from pairwise comparison itself. A simulation shows why, and why BT is still the better default: with binary quality (correct or wrong), as on our benchmarks, BT and Borda select equally well, while with graded quality, as in heuristic optimization tasks, BT consistently picks slightly better candidates. We revise the paper accordingly.

Q3. How does OpenDeepThink compare with Duel-Evolve, V1, and Squeeze Evolve under matched inference budgets?

Thank you for the suggestion. Within our setting:

  • V1: used as our final selector on the same pools, V1-Infer performs on par with BT (74.4% vs 71.8%; the CI includes zero), but needs 6 sequential batches instead of one.
  • Duel-Evolve: no public code, and its depth is far beyond our setting.
  • Squeeze Evolve: it allocates work across models of different cost, which is orthogonal to our single-model setting; we discuss it in Section 2.

Q4. Can the authors provide more extensive evaluations on non-Gemini models and explain the performance degradation on HLE?

Thank you; we added both.

  • DeepSeek V4 Flash (open-weight), 30 problems: Elo +214 [78, 359], versus +392 for Gemini 3.1 Pro on the same problems.
  • Qwen3.5-9B, 4 problems: pass@1 rises from 20% to 56%, and the final pick is correct on all 4; judge reliability is the bottleneck at this scale.
  • HLE: on mixed pairs, the judge picks the correct answer only 58.7% of the time (88.6% on programming), so selection amplifies noise. We replace the per-domain explanation with this measurement.

Thank you again for your review. We would be grateful if you could check our response and the revised PDF before the discussion ends; we also added an end-to-end comparison with the full V1 pipeline (see our reply to Reviewer Vvqi).

Thank you for the clarification. I appreciate the authors acknowledging that Bradley–Terry (BT) aggregation is not necessary for the main performance gains. However, given that the paper places substantial emphasis on BT, this finding raises further concerns about the methodological novelty and positioning of the work. The additional simulation suggests that BT may be preferable for graded-quality tasks, but this advantage is not demonstrated on the evaluated benchmarks. I encourage the authors to reconsider the emphasis on BT and clarify the actual methodological contributions

Reviewer_245xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OpenDeepThink, a training-free test-time compute framework that combines pairwise LLM judgment, Bradley–Terry aggregation, and evolutionary search for parallel reasoning. It iteratively generates candidate solutions, ranks them through pairwise comparisons, and uses natural-language critiques to guide mutation while eliminating weaker candidates.

Evaluated on 192 competitive programming problems and 82 HLE questions, OpenDeepThink improves Gemini 3.1 Pro's effective Codeforces Elo by 405 points. On hard problems, pass rate increases from 11% to 36%, while BT top-1 accuracy rises from 23% to 50%. The paper further investigates cross-model transfer, comparison strategies, feedback mechanisms, and cross-domain generalization, revealing limitations in domains with less reliable LLM judgments.

Critical strengths
  1. A practical framework for parallel test-time evolution. The paper addresses an important bottleneck in parallel reasoning: selecting and improving candidates without relying on external verifiers. Combining pairwise comparisons with natural-language critiques provides both a ranking signal and actionable feedback for candidate mutation. The synchronous execution design also reduces sequential reasoning depth, which is practically relevant for inference-time scaling.

  2. A useful empirical investigation of pairwise judgment and feedback. The experiments cover 192 competitive programming problems and include comparisons against pointwise judgment and Self-Refine, as well as feedback ablations and judge-prompt robustness analyses. The results provide evidence that pairwise comparison can be more reliable than independent pointwise evaluation, and that negative comparative feedback can improve candidate generation.

Critical weaknesses
  1. Insufficient comparisons with closely related methods. Although the paper compares against pointwise judging and Self-Refine, it lacks direct comparisons with closely related pairwise preference-based evolutionary approaches under matched models and computational budgets. Moreover, the ablation between Bradley–Terry and Borda count shows nearly identical performance, with identical top-1 selections on 186 out of 192 programming problems. This makes it difficult to establish the practical advantage of the proposed aggregation strategy over simpler alternatives.

  2. Insufficient ablation of key design choices. Several important design decisions, including the 25% elite retention ratio and the license-to-abandon prompt, are selected through informal tuning rather than controlled ablation. This leaves their individual contributions and the sensitivity of the framework to these choices insufficiently understood.

  3. Limited model and benchmark coverage. The evaluation is primarily conducted on Gemini-family models (Gemini 3.1 Pro, Gemini 3 Flash, and Gemini 2.5 Pro), while the non-Gemini evaluation on DeepSeek V4 Flash covers only five problems. In terms of benchmarks, the main evaluation focuses on competitive programming. Although HLE provides some cross-domain evaluation, it includes only 82 questions, which is insufficient to establish the method's applicability across a broader range of reasoning tasks.

  4. Limited evidence for generalization and computational efficiency. The overall BT top-1 accuracy on HLE decreases after applying the proposed framework, suggesting that its effectiveness depends substantially on the reliability of the underlying LLM judge and the task domain. Furthermore, the framework requires approximately 285 API calls per problem. While the sequential depth is relatively small, the actual computational cost, token consumption, and end-to-end latency under realistic concurrency constraints are not sufficiently characterized.

Questions for authors
  1. Comparison with related methods: How does OpenDeepThink differ from closely related pairwise preference-based evolutionary approaches, particularly Duel-Evolve? Do the authors have results comparing these methods under matched models, benchmarks, and computational budgets?

  2. Contribution of Bradley–Terry aggregation: Table A.1 shows that Bradley–Terry and Borda count achieve identical top-1 accuracy on the programming benchmarks, with identical selections on 186 out of 192 problems. Does BT provide benefits beyond final candidate selection, particularly in guiding evolutionary mutation? Can the authors provide further evidence of its contribution?

  3. Cross-model generalization: Have the authors evaluated OpenDeepThink on additional open-weight models, particularly smaller models (e.g., 7B–14B) from different model families? If so, do the improvements remain consistent? This would help clarify whether the framework generalizes beyond strong Gemini models.

  4. Computational efficiency: Can the authors provide actual token consumption, API costs, and end-to-end wall-clock latency under realistic parallel execution settings? In particular, how does OpenDeepThink compare with Self-Refine when matching token budgets rather than only API call counts?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 245x

Thank you for the detailed questions. Numbers below use the corrected data (see General Response).

Q1. How does OpenDeepThink differ from closely related pairwise preference-based evolutionary approaches, particularly Duel-Evolve?

Thank you for this question; the distinction matters, and we now make it explicit in Section 2. Duel-Evolve runs 150–200 adaptive generations and, on code, feeds public-test execution traces to its generator and judge. OpenDeepThink uses no tests at all and a fixed depth of 8 sequential rounds, which leads to fixed pairings run in parallel, a fixed population with elites, and full pairwise critiques as the mutation signal. Duel-Evolve has no public code, so we compared with V1-Infer, used as our final selector on the same pools: it performs on par with BT (74.4% vs 71.8%; the CI includes zero) but needs 6 sequential batches instead of one.

Q2. Does BT provide benefits beyond final candidate selection, particularly in guiding evolutionary mutation?

This is a good question, and we checked it directly. We found no measurable benefit: BT and Borda keep equally good candidates during evolution and select similarly at the end, and both beat raw win rate when comparisons are sparse. What matters is the pairwise ranking itself: on Hard, culling the bottom quarter removes only 7.6% of correct solutions, versus 25% for random culling. On the design choices you name: elite replacement keeps every correct elite (vs 93.8% without it), and removing the license-to-abandon sentence has no detectable effect.

Q3. Have the authors evaluated OpenDeepThink on additional open-weight models, particularly smaller models (e.g., 7B–14B) from different model families?

Thank you for the suggestion; we added two models from other families.

  • DeepSeek V4 Flash (open-weight), 30 problems: Elo +214 [78, 359], versus +392 for Gemini 3.1 Pro on the same problems.
  • Qwen3.5-9B, 4 problems where it has partial competence: pass@1 rises from 20% to 56%, and the final pick is correct on all 4. As a judge it often runs out of reasoning budget, which is the bottleneck at this scale.

On HLE, the judge is close to chance on mixed pairs (58.7%), and the method does not help; we state this as the main limitation.

Q4. Can the authors provide actual token consumption, API costs, and end-to-end wall-clock latency under realistic parallel execution settings?

Thank you; we will report these measurements in the paper. With thinking tokens logged, OpenDeepThink costs about 103 USD per Hard problem, versus 96 USD for Self-Refine with pointwise selection, so the comparison is roughly cost-matched. The ideal wall-clock time is about 28 minutes (8 rounds); measured time was limited by API throughput, not by depth.

Thank you for the response and additional experiments. I appreciate the authors' efforts to address my concerns. The new ablations and cross-model evaluations strengthen the paper. However, some concerns remain, particularly the lack of an end-to-end comparison with closely related methods, the limited evidence for broader generalization, and the absence of measured wall-clock latency.

Thank you for the reply. We tried to address the remaining concerns below.

the lack of an end-to-end comparison with closely related methods

We ran the full V1 pipeline (sample 20 solutions, then the V1 tournament) on the same initial samples. Results on the 62 Hard problems with Gemini 3.1 Pro:

MethodCallsCost per problemTop-1
Best-of-20 + V1-Infer (V1 pipeline)80~28 USD29.0%
Self-Refine + pointwise300~96 USD42.8%
Self-Refine + pairwise BT240~85 USD51.6%
OpenDeepThink285~103 USD50.0%
OpenDeepThink, V1-Infer as final selector245~89 USD51.6%

OpenDeepThink is 21.0 points higher than the V1 pipeline (95% CI [11.3, 30.6]). The main reason is that half of the Hard problems have no correct initial sample, so selection alone cannot solve them. When V1 selects from our evolved pool instead, it also reaches 51.6%.

the absence of measured wall-clock latency

We measured it from the call logs. Each round has to wait for its slowest call, so the real time is longer than the 28 minutes we wrote before: about 75 minutes per Hard problem (about 38 minutes for the V1 pipeline). We also did a live run with DeepSeek V4 Flash on 5 problems, sending all calls of a round at once: OpenDeepThink took 61 minutes and Self-Refine took 72 minutes. We updated these numbers in the revised PDF.

the limited evidence for broader generalization

We agree this is the main limitation. Our method needs a judge that can tell correct and wrong solutions apart. On programming, the judge is correct in 88.6% of such pairs, but on HLE only 58.7%, and our method only helps in the first case. We now state this scope clearly in the paper.

Reviewer_r5QXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenDeepThink scales test-time compute without external verifiers through parallel sampling and multi-generation evolution. In each generation, the same model compares randomly paired candidate solutions, retains the top 25% unmodified and discards the bottom 25% by Bradley–Terry (BT) scores, and rewrites each of the top 75% using critiques generated from pairwise comparisons. A final denser pairwise round selects the submitted solution. Experiments cover 192 competitive programming problems and 82 HLE questions, and the authors release the CF-73 benchmark.

Critical strengths
  • Clear and practical pipeline. After parallel sampling, the same model supplies pairwise judgments and performs rewriting, while BT scores determine filtering. In the main coding experiments, the sequential depth is 8 rounds, requiring no external verifier at inference time and reducing wall-clock time under sufficient parallelism.
  • Strong absolute results on competitive programming. OpenDeepThink substantially improves over single-pass sampling on Hard problems. The curated CF-73 local judge reportedly achieves 99% agreement with official verdicts, aiding future reproduction.
Critical weaknesses
  • The BT-specific advantage is unsupported, and the pointwise comparison is not isolated. At K = 4, every candidate plays the same number of matches, so Borda is equivalent to fractional win rate. Table A.1 reports identical BT/Borda top-1 accuracy at K = 4 (72.3%) and M = 10 (82.8%), with 186/192 identical final selections and confidence intervals spanning zero. Thus, this ablation detects no top-1 accuracy benefit from BT over Borda in these settings. The 41% vs. 50% Self-Refine comparison also changes both the candidate pool and selector, so the gap cannot be attributed purely to selection; cross-selection is needed. In addition, the baseline merges feedback and rewriting rather than following the original Self-Refine protocol.
  • The feedback ablation does not support the claim that critiques drive nearly all mutation gains. Relative to no feedback, K = 4 critiques add only 24 net transitions, accounting for 24/68 = 35% of the reported net gain. The ratio 68/44 = 1.55× is also not "nearly double." Confidence intervals, cluster-aware significance tests, and exact definitions of several feedback variants are missing.
  • Superiority over simpler or related methods at comparable compute is not established. The ~300-call Self-Refine baseline is tested only on Hard problems and changes both generation and selection. Matching API-call counts also does not match tokens, FLOPs, cost, or latency because pairwise and mutation calls contain substantially different context. On CF-73, accuracy is 96% at gen-1 and at the end. The 500-pair diagnostic compares pointwise joint correctness (59.2%) with pairwise accuracy (86.2%) without defining how pairwise ties enter the latter; the rates in Section A.3 imply 79.3% expected pointwise selection accuracy under random tie-breaking. The "27 vs. 1 uniquely solved" comparison is likewise asymmetric, contrasting a realized BT top-1 outcome with certainty across all tied-top baseline candidates. Common selection metrics are needed. Comparisons with stronger pointwise or cited related pairwise methods, and ablations of the elite/cull ratio and license-to-abandon instruction, are missing.
  • Several results have confounded attribution, and HLE lacks reproducibility details. Table 2a tests each model on a different difficulty subset, so model capability is confounded with difficulty. The +27 cross-generation BT increase is not the selector's marginal contribution; Section 4.2 instead decomposes the Hard gain into +25 from evolution and +14 from selection. The marginal gain of the final dense round is 3.1 points (46.9% to 50.0%); the claimed 8-point gain relative to sparse gen-2 also includes the third evolution round. For HLE, changing n and T to 12 and 2 contradicts Section 4.1's statement that all four hyperparameters are fixed across benchmarks, and the prompts, grading, and majority-vote procedure are omitted. The category-level reliability explanation is based on small samples and post-hoc accuracy changes rather than direct judge evaluation.
  • Pipeline-level uncertainty is underreported. Table A.2 shows only 74.2% agreement when the temperature-0 judge is rerun, yet no repeated-pipeline or random-seed uncertainty is reported for the main results. Bootstrap intervals over problems do not capture variation from generation, random pairing, or temporal judge drift.
Questions for authors
  • What evidence supports BT over Borda, and how do the Self-Refine and OpenDeepThink pools perform under the same selectors?
  • How are TIEs scored in the 500-pair diagnostic, and how do the two judges compare under the same selection metric? How does the "uniquely solved" comparison change under symmetric tie handling? Are the Table 3 differences statistically significant under a problem-clustered analysis?
  • Why were n and T changed for HLE, what prompts and grading pipeline were used, and why do Tables 2b and A.1 report 54.3% and 54.9%?
  • How do methods compare in input/output tokens, cost, and measured latency rather than API-call count, and how stable are the main results across independent pipeline runs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer r5QX

Thank you for the careful review. Numbers below use the corrected data (see General Response).

Q1. What evidence supports BT over Borda, and how do the Self-Refine and OpenDeepThink pools perform under the same selectors?

Thank you for raising this; it is a central question for the paper. We agree that BT gives no clear benefit over Borda: the two perform similarly throughout, and both beat raw win rate when comparisons are sparse. We now present BT as a sound aggregator rather than a source of gains, and clarify that Table A.1 counts problems with the same correctness (identical selections: 84).

A simulation with a noisy judge explains this and still supports BT as the default. When solution quality is essentially binary (correct or wrong), as on our benchmarks, BT and Borda select equally well. When quality is graded, as in heuristic optimization tasks, BT's correction for opponent strength consistently picks better candidates (about 2 points more often in the top 3 with our random pairing). We will add this simulation to the appendix.

We ran the cross-selection on the 62 Hard problems. Self-Refine pools reach 42.8% with pointwise selection and 51.6% with BT; OpenDeepThink pools with BT reach 50.0%. So most of the gap to Self-Refine comes from selection (+8.8 [1.5, 15.9]), not from the pool (−1.6, n.s.). Evolution still produces better pools (40.8% vs 30.6% correct candidates), which helps most when few comparisons are available. On the +27: the split depends on the order of attribution (+31/+9 or +26/+14.5), and we report both. We also state that our Self-Refine merges feedback and rewriting.

Q2. How are TIEs scored in the 500-pair diagnostic … "uniquely solved" … Are the Table 3 differences statistically significant?

Thank you for the careful reading of these diagnostics. The raw 500-pair records were not kept, and your 79.3% estimate is right. We replace the diagnostic with a like-for-like comparison on 1,706 correct-vs-incorrect pairs under one tie rule: one pairwise call is correct 74.4% of the time, versus 63.2% for one pointwise call per side, and pointwise needs 16 calls to catch up. With symmetric tie handling, "uniquely solved" is about 8 vs 3.5.

Table 3 is replaced by a paired control (the same parents mutated with and without feedback): feedback raises the rescue rate from 14.2% to 18.7% (+4.5 [1.7, 7.4]). We drop "nearly double". Removing the license-to-abandon sentence has no detectable effect, and elite replacement keeps every correct elite.

Q3. Why were n and T changed for HLE, what prompts and grading pipeline were used, and why do Tables 2b and A.1 report 54.3% and 54.9%?

Thank you for catching these inconsistencies. We reduced n and T on HLE to limit cost; we now state this and drop the claim of fixed hyperparameters. The two numbers used different denominators and graders. With one grader, BT top-1 goes from 62.2% to 54.9%, so the method does not help on HLE. We replace the per-category explanation with a direct measurement: the judge picks the correct answer in 58.7% of mixed pairs on HLE, versus 88.6% on programming. We will add the prompts, grading and majority vote to the appendix.

Q4. How do methods compare in input/output tokens, cost, and measured latency … and how stable are the main results across independent pipeline runs?

This is an important practical point. With thinking tokens logged, OpenDeepThink costs about 103 USD per Hard problem, versus 96 USD for Self-Refine with pointwise selection and 85 USD with BT, so costs are of the same order. The ideal wall-clock time is about 28 minutes; measured time was dominated by API throughput limits. Full repeated runs are beyond our budget. A rerun of the final selection with fresh judge calls, relevant to Table A.2, is in progress; we will post it in this thread.

Other points. We remove cross-model claims from Table 2a, since each model was evaluated on a different tier. The dense final round adds 1.6 points on the corrected data, and we fix Table A.4. CF-73 is near ceiling by gen-1, consistent with most rescues happening in the first generation; we will state this.

Thank you again for the review. Here are two updates related to your questions.

how stable are the main results across independent pipeline runs?

We reran the final selection with new judge calls and a new seed on 27 mixed Hard pools. It picks a correct solution on 18/27, compared with 17/27 in the original run (+3.7 pp, 95% CI [−7.4, 14.8]). The chosen solution is often different, but the accuracy is stable.

measured latency

The 28 minutes in our reply was only a lower bound. Because each round waits for its slowest call, the measured time is about 75 minutes per Hard problem. In a live run with all calls sent at once, OpenDeepThink took 61 minutes and Self-Refine took 72 minutes. We corrected this in the revised PDF.

Thank you again for your review. We would be grateful if you could check our responses and the revised PDF before the discussion ends, and let us know if any concerns remain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. Below we summarize the main changes; the per-reviewer replies give details.

Data audit. While preparing the rebuttal, we re-audited all results against the raw logs. We re-ran a few early problems with the final code and recomputed the Self-Refine baseline and HLE grading consistently; this also moves three problems between difficulty tiers. The headline results are unchanged (+405 Elo; 50.0% top-1 on Hard). On HLE, consistent grading shows no improvement, and we now present HLE as a limitation. Two diagnostics whose raw logs were not kept are replaced by new controlled experiments. All numbers in our replies use the corrected data, and Appendix A.1 lists every correction.

Main points of the revision.

  • Setting. OpenDeepThink uses no tests of any kind and a fixed sequential depth of 8 LLM rounds.
  • Evolution improves the pool. On Hard, the share of correct candidates rises from 9.7% to 40.8%.
  • Pairwise selection is what makes the choice work. It clearly beats pointwise selection, and most of our advantage over Self-Refine comes from it. The aggregator matters little: BT and Borda perform similarly, and V1-Infer used as our final selector performs on par with BT.
  • Critique feedback adds a moderate but significant gain.
  • Limitation. The method needs a judge that can tell right from wrong; on HLE it cannot (58.7% on mixed pairs, vs 88.6% on programming).

A rerun of the final selection with fresh judge calls is in progress; we will post it here. We have uploaded a revised PDF with all changes marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.