- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tries to understand when more reasoning is needed / when more reasoning flips a correct answer wrong. The paper proposes an auditing / verdict (?) framework to detect when the answer would flip under more reasoning. The paper then verified the method on one dataset (Math500) and two very old deepseek models. Overall I don't quite understand what the paper is doing, there are too many concepts I've never learned about before: verdict conversatoin / harmonic reference / martingale
Critical strengths
-
The problem studied in the paper is somewhat important: Understanding when to stop the reasoning model is a problem widely studied in 2025.
-
The experimental setting is well documented.
-
There are lots of equations and theorems; Although I don't understand them, the method seems to be very theoratically-grounded.
Critical weaknesses
-
I am not sure who the target audience is: I have trouble understanding sections 3 and 4 as well as most of the paper. The paper seems to be more suitable for journals as the paper is very theory-heavy. I would not be surprised if reviewers would be tempted to just feed the draft into LLM for assistance in understanding.
-
The evaluation is limited to a single dataset Math500 and some old DeepSeek models from early 2025.
-
Some empirical results are hard to understand, e.g. what's survivorship in Fig.2
Questions for authors
Can the author explain the paper in plainer language what they problem they are trying to resolve? Or what's the background knowledge is required to understand the paper? I have worked on reasoning models and efficient early stopping of reasoning models, but I can't even understand Figure 1.
Why is Qwen 7B model using NF4... why is precision even specified or changed?
4 comments
Follow-up to Reviewer YSLt
Dear Reviewer YSLt,
Thank you again for your review. As the author–reviewer discussion closes on November 18, we would like to check whether our response and the revised PDF address your concerns. In brief:
- Clarity. The revision states the problem in early-stopping terms: a probe of final correctness cannot tell whether stopping now helps, so two quantities are needed. The new Fig. 1 shows both comparisons with a worked example.
- Second benchmark and a 2026 model. On OlympiadBench, harmful overthinking occurs in 3.8% of chains, and the chains still reasoning improve (+7.3, 95% CI [+3.1, +11.8]) while composition pulls accuracy down. Qwen3.5-2B (released February 2026) shows 1.9% and the same pattern.
- Precision. The 7B model in 8-bit gives 3.1% against 3.4% in NF4, with almost the same decomposition; switching the 1.5B model to NF4 also leaves the results unchanged.
If any point is still unclear, we would be glad to address it before the deadline. If your concerns are resolved, we would be grateful if you would consider updating your score.
I've read the author's response and a glance over the updated draft, which seems to be a complete rewrite.
I've decided to maintain my score as there is still huge room for the presentation to be improved.
I encourage the authors to remove all unncessary math and hard-to-interpret terminology as they could be too difficult and time-consuming for human readers to understand.
Response on presentation
Thank you for taking another look at the revision.
We agree that the paper should be easy to follow for readers working on reasoning models, and the revision is organized around that. Each formal result in Secs. 2–3 is followed by a plain statement of what it means for the decision to stop. Fig. 1 works through both comparisons with a small example, and the main empirical findings are shown directly in Figs. 2–5. A notation glossary is in App. A.
At the same time, the paper is a statistical analysis of LLM reasoning, so its main results are formal statements: a forecast of final correctness has no predictable change (Prop. 2), the commitment–verdict gap equals the value of stopping (Cor. 4), and accuracy-versus-budget curves decompose into two terms (Prop. 5). These statements define what the experiments measure, so we keep them in the main text and place proofs in the appendix. The same tools are common in this line of work at ICLR and related venues, for example:
- early exit with statistical guarantees: Schuster et al., Confident Adaptive Language Modeling (NeurIPS 2022);
- statistical guarantees for LLM outputs: Quach et al., Conformal Language Modeling (ICLR 2024); Angelopoulos et al., Conformal Risk Control (ICLR 2024);
- value estimates from Monte Carlo rollouts for reasoning: Lightman et al., Let's Verify Step by Step (ICLR 2024); Wang et al., Math-Shepherd (ACL 2024);
- allocating test-time compute: Snell et al. (ICLR 2025).
For background, the stopping comparison follows Chow, Robbins and Siegmund, Great Expectations: The Theory of Optimal Stopping (1971), and Williams, Probability with Martingales (1991) introduces the martingale tools we use.
If a specific passage remains hard to follow, we would be glad to clarify it here or in the final version.
Response to Reviewer YSLt
Thank you for the clear questions. The revision is rewritten for readers who work on reasoning models and early stopping, as you do. Below we explain the problem in that language, then answer your questions about Figs. 1–2, the datasets and models, and NF4.
What problem are we solving? Many early-stopping methods train a probe that predicts whether the model's final answer will be correct, and stop when the probe is confident. Our point is that this probe predicts the wrong thing for deciding when to stop.
The only background required is conditional probability. "Martingale" means only that an accurate forecast cannot predict its own next revision. That is why a steady forecast of final correctness says nothing about whether reasoning is helping. Terms such as "harmonic reference" are now in the appendix, and the main text uses just these two quantities.
Figure 1. The revision replaces it with a worked example for each of the two comparisons (new Fig. 1).
Survivorship in Fig. 2 (Fig. 3 in the revision).
A single dataset and older models.
Why NF4 for the 7B model, and why state the precision? We loaded the 7B model in NF4 for memory efficiency. NF4 is the standard 4-bit format introduced with QLoRA (Dettmers et al., 2023) and designed to preserve 16-bit quality. We state the precision because our two main settings differ in it, and we tested directly that it does not affect the results.
We hope the plain-language explanation, the new figure and the added benchmark and model make the paper's problem and evidence clear. If so, we would be grateful if you would consider raising your score.