- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how different methods for producing final answers from reasoning prefixes affect comparisons between early-stopping techniques. As I understand it, no method consistently performs best, and the optimal choice depends on the budget allocated to generating the final answer.
Critical strengths
None
Critical weaknesses
The paper is very difficult to follow. Despite being very familiar with the area, I couldn't understand the paper's contribution from the abstract.
The experimental design is also hard to follow, even though the experiments are simple in nature.
While I believe that careful and accurate comparisons of early-stopping techniques and their combinations with methods for producing final answers are important, I think that the current revision of this paper fails to communicate any important findings.
The presentation makes it difficult to distinguish the main findings from other conclusions.
Questions for authors
I don't have questions whose answers could materially affect my assessment.
Clarifying the main finding and experimental design
We have revised the paper around one question: why can the same stopping decisions yield different measured accuracy losses? The comparison below uses the same 200 Qwen3.5-9B/AMC23 runs, stopping decisions, scorer and 1,024-token answer allowance. Only the answer procedure changes. AC here is the blank-line adaptation of Answer Convergence.
The reported stopping loss falls by 28.5 percentage points (simultaneous 95% CI [12.5, 44.5]). Of this reduction, 20.5 points come from better answers under the AC policy, while 8.0 come from a lower Full score. Both procedures keep naturally completed answers, so Full changes only on runs that reach the reasoning limit and still require an answer. A smaller stopping loss can therefore partly reflect a weaker reference answer, even though the stopping decisions are unchanged.
Pairing each cut with its own full trajectory also lets us isolate the change at the cut. Among the 92 interrupted runs that would have ended naturally, Full is identical under the two procedures, and native closure gives a net 29 additional correct answers at the cuts (Sections 2.1–2.2).
We also tested two closed 2K procedures that meet the 95% validity threshold at Full, under each policy and among triggered stops. Both retain a small estimated AC lead over the length-matched fixed cut, while the estimated Full−AC loss changes from 5.5 to 8.0 pp (exploratory pointwise 95% CI for the paired change [0.0, 5.5], Table 2 and Appendix C.2). This separates a change in the measured stopping loss from a change in which rule ranks higher. We recommend reporting both endpoint accuracies on the same runs, stating whether naturally completed answers were kept or regenerated, and counting probes and answer generation when comparing computational cost.
We have reorganized the paper around this argument. The revised title, What Does the Accuracy Loss from Early Stopping Measure?, and abstract state this question. Figure 1 displays the four scores on the first page, and Section 2.1 works through them before introducing the equation. Section 3 examines when a stopped trace can still yield a correct answer, Section 4 compares rule rankings, and Section 5 accounts for system cost.