- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper explores what to do when giving a language model a fixed inference budget: should it think longer in a single generation, try multiple independent attempts, or have multiple agents repeatedly revise and discuss. The authors propose an adaptive method and run some experiments to validate it, along with some trivial theoretical analysis.
Critical strengths
The author proposes their own method and conducts a set of experiments, supported by several figures illustrating the approach and results. Additional detailed results are provided in the appendix.
Critical weaknesses
-
The paper's theory and modeling have major issues. The authors use a lot of complex mathematical notation in places where it is completely unnecessary. At the same time, in places that genuinely require mathematical understanding, they fail to do so, demonstrating a real lack of familiarity with the field. In Line 140, the authors discuss reasoning cost, but they only consider tokens and completely fail to factor in FLOPs as an approximation for actual compute.
-
The proposed method is not novel; plenty of previous work has already addressed this much more comprehensively.
-
The method requires probing every model a certain number of times for each problem to decide which approach to use. The authors should consider this cost.
-
The scope and scale of the experiments are far too small to support their claims.
Questions for authors
NA
3 comments
Hello! As the deadline is approaching, we'd be greatful if you could find the time to comment on this :) We genuinely believe that we could be actionable about your comments/weaknesses if you could be more precise. Gives us a change to convince you the worth of our paper :)
Good luck with your own submissions,
Best,
The authors
Thank you for the response. The new probe-cost experiment shows that fixed depth (70.5%) outperforms the probe-based allocator (64.0%) once probe costs are charged, which reinforces my concerns. I will keep my score.
Response to Reviewer j9dP
Thank you for your review. We address the compute and probing concerns below and request the specific information needed to assess the remaining objections. Brackets denote 95% confidence intervals.
1. Mathematical presentation and soundness.
You state that our theory and modeling have “major issues.” Apart from the compute-accounting concern addressed below, however, the review does not identify a mathematical statement, assumption, inference, or proof step alleged to be incorrect.
As per ICLR reviewer guidelines can you “Provide supporting arguments for your recommendation.” It would help us addressing if you could identify each mathematical issue and how it affects our conclusions. Likewise, please specify which mathematical details you consider unnecessary and which phenomena you believe require analysis that is absent.
These particulars are necessary for us to evaluate and address the stated soundness concern. We are ready to examine each specific issue and make any correction it warrants as soon as we know!
2. Reasoning tokens versus FLOPs.
The manuscript explicitly states in §8, lines 385–386:
The distinction between reasoning-token expenditure and total computation is therefore explicitly recognized. The OpenRouter runs used for the main experiments did not provide FLOP telemetry, and the paper states the resulting limitation clearly in lines 385-386.
To address the substantive compute question, we have now added 4,992 calls comparing depth, voting, and debate on dense Qwen3-8B and Qwen3-32B under natural stopping, with estimated FLOPs accounting for both prefill and decoding (Experiments 4–5). Depth is more accurate while voting and debate use 1.68–2.93× as many estimated FLOPs, or 1.57–2.68× with shared-prefix caching.
Thus, for these dense models, accounting for prefill and decoding preserves the ordering: the less accurate alternatives also consume more estimated computation. If you consider this comparison insufficient to address your objection, please identify the missing cost term or the specific comparison that remains inadequately controlled.
3. Novelty relative to prior work.
The ICLR reviewer guidelines require to substantiate your statement that previous work addresses our contribution more comprehensively, identifying the relevant papers and specifying which of our contributions they already establish. We apologize if we have overlooked relevant work and will correct any missing attribution. However, in our knowledge our contributions do not appear in the related literature.
4. Per-problem probe cost.
The manuscript explicitly states in §6, lines 322–323, that “probes add 5A to solver cost,” and distinguishes the reported tradeoff from an equal-total-compute comparison.
The supporting experiment demonstrates predictive signal on held-out problems: the additive model, which includes probe and provider features, achieves a Brier score of 0.179 versus 0.202 for the population-only fit (§E.2). Each routing decision uses five 1,024-token solver calls, adding 5,120 reasoning tokens, approximately USD 0.018, and 33 seconds of modeled latency.
We have now charged all five probes against a shared 16,384-token ceiling. On 200 held-out problems, fixed depth achieves 70.5% accuracy versus 64.0% for the allocator, a 6.5-percentage-point advantage [1.5, 11.5] (Experiment 2). This reports the consequence of paying for the probes within the available budget, including the resulting advantage for fixed depth.
Can you please tell us any component of the per-problem probing cost that you believe remains unaccounted for?
5. Experimental scope and scale.
You state that the experiments are “far too small to support their claims.” Please specify which claim you consider unsupported and what experimental scope would provide compelling evidence for that claim?
The expanded evaluation covers seven models from five families, 96 problems from four benchmarks, and two repeats per item, together with additional 4,096–65,536-token budget sweeps on DeepSeek and Claude. We believe this substantially strengthens the evidence for the conclusions within the evaluated settings.
We would appreciate if you could reassess the scope objection against this expanded evaluation. If you maintain it, can you please identify the remaining gap? E.g., model coverage, task coverage, sample size, or a particular comparison. Also, would you be willing to explain which conclusion depends on the additional evidence you would like to see. This would allow us to address concretely!
We hope you update your assessment in light of these clarifications and additional results, and state the specific grounds for any concerns you retain. We remain ready to address those grounds directly :)