- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents OpenDeepThink, a framework for scaling LLM reasoning through parallel sampling without an external correctness verifier. The same model generates and compares candidate solutions, using Bradley–Terry rankings for selection and comparison critiques for iterative refinement. Experiments primarily focus on competitive programming, showing improvements over the base model and the implemented baselines.
Critical strengths
- The paper identifies candidate selection as a key challenge in parallel reasoning without an external verifier. It explains how pairwise comparison, ranking, and feedback-guided refinement fit together, and Algorithm 1 makes the overall procedure straightforward to understand.
- Beyond the main results on two programming benchmarks and HLE, the experiments examine how performance changes across generations and how different feedback strategies affect refinement.
- The same hyperparameters are used for Gemini 3.1 Pro, Gemini 3 Flash, and Gemini 2.5 Pro, with improvements in both candidate quality and final selection accuracy, suggesting that iterative refinement adds value beyond selecting from the initial samples.
Critical weaknesses
- The baseline comparisons are not strong enough. The related work discusses several closely related test-time scaling methods, including Duel-Evolve and V1, but neither is evaluated directly. Instead, the comparisons focus on pointwise selection and a simplified Self-Refine implementation. These results show improvements over relatively simple alternatives, but do not establish what the proposed pipeline offers over the closest existing methods.
- The paper reports API-call counts, a latency estimate, and a BT-only scaling study, but lacks an end-to-end comparison of accuracy against token usage and measured wall-clock time across methods. Matching call counts does not necessarily match cost, since generation, comparison, and refinement have different token requirements. Given that efficiency is a central motivation, better accuracy than simple baselines alone is not enough to show that this is a better use of test-time compute.
- The pipeline requires a population size, comparison budgets, an iteration count, and an elite ratio. While some settings are examined, the full pipeline lacks a systematic sensitivity analysis, and the elite ratio was chosen through informal tuning. This leaves practical questions about how to allocate a new budget or adapt the method to a new task without costly trial and error.
Questions for authors
-
Could you expand the comparisons to include stronger, closely related test-time scaling baselines? The current pointwise and modified Self-Refine baselines do not establish the pipeline’s advantage over existing approaches discussed in §2.
-
With these additional baselines included, how does your method compare in accuracy versus total token usage and measured end-to-end latency under comparable concurrency limits? Such comparisons would help establish whether the framework offers a better cost-performance trade-off, rather than simply higher accuracy.
-
How sensitive is the full pipeline to population size, comparison budgets, iteration count, and elite ratio? What guidance can you offer for choosing these settings, and deciding when to stop, on a new task without access to correctness labels?
4 comments
Thank you for the detailed response and the additional experiments. I appreciate the effort, but my main concern remains. The V1-Infer result compares only the final selector on the same candidate pools, so it does not show how the full OpenDeepThink pipeline compares with other test-time scaling methods. I would also like to see these methods compared end to end under the same setup, especially in terms of measured wall-clock time, token usage, and accuracy. This is important because parallel efficiency is a key motivation of the paper. The additional tuning advice is helpful, though the number of manually chosen hyperparameters may still make the method difficult to use on new tasks.
Thank you for the reply.
The V1-Infer result compares only the final selector on the same candidate pools … compared end to end under the same setup, especially in terms of measured wall-clock time, token usage, and accuracy.
We agree. We ran the full V1 pipeline (sample 20 solutions, then the V1 tournament) on the same initial samples. Results on the 62 Hard problems with Gemini 3.1 Pro:
| Method | Calls | Cost per problem | Top-1 |
|---|---|---|---|
| Best-of-20 + V1-Infer (V1 pipeline) | 80 | ~28 USD | 29.0% |
| Self-Refine + pointwise | 300 | ~96 USD | 42.8% |
| Self-Refine + pairwise BT | 240 | ~85 USD | 51.6% |
| OpenDeepThink | 285 | ~103 USD | 50.0% |
| OpenDeepThink, V1-Infer as final selector | 245 | ~89 USD | 51.6% |
The V1 pipeline is cheaper and faster (about 38 vs. 75 minutes per problem, measured from call logs), but it can only select from the initial samples, and half of the Hard problems have no correct initial sample. OpenDeepThink is 21.0 points higher (95% CI [11.3, 30.6]). We also did a live run with DeepSeek V4 Flash on 5 problems, sending all calls of a round at once: OpenDeepThink took 61 minutes and Self-Refine took 72 minutes. The 28 minutes in our first reply was only a lower bound, and we corrected it in the revised PDF.
the number of manually chosen hyperparameters may still make the method difficult to use on new tasks
We used the same default setting for all models (three Gemini models, DeepSeek V4 Flash and Qwen3.5-9B) without any retuning, so we expect the defaults to be a reasonable starting point for a new task. Also, the label-free stopping rule from our first reply can end a run early on easy problems and save calls.
Thank you again for the discussion. We hope the end-to-end comparison addresses your main concern, and we are happy to clarify anything before the discussion ends.
Response to Reviewer Vvqi
Thank you for the practical questions. Numbers below use the corrected data (see General Response).
Thank you for the suggestion. We compared with V1-Infer (public code), used as our final selector on the same pools with the same judge model. It performs on par with BT (74.4% vs 71.8% on 39 mixed Hard pools; the CI includes zero), but needs 6 sequential batches instead of one. Duel-Evolve has no public code, needs 150–200 sequential generations, and uses public-test execution on code, which our setting excludes.
This is an important point, since efficiency is a central motivation of the paper. With thinking tokens logged (over 90% of cost), on Hard:
Costs are of the same order, and pairwise selection is what lifts accuracy. The ideal wall-clock time is about 28 minutes (8 rounds); in practice, time was limited by API throughput, not depth.
Thank you for asking; we will add concrete guidance to the paper.
Recipe for a new task. Default: n = 20, K = 4, T = 3, 10 final rounds (about 285 calls). Under a tight budget, a lighter setting such as the one we used on HLE (n = 12, T = 2, 6 final rounds; about 115 calls) applies the reductions above together; we validated each reduction on programming, but not this exact combination. In both cases, stop early at 75% ties. We will add this guidance to the paper.